2021-11-20T07:20:12.039 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-20T07:20:12.041 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi --archive /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123 --verbose -- /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/orig.config.yaml 2021-11-20T07:20:12.101 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-11-20T07:20:12.181 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123 branch: wip-yuri2-testing-2021-11-19-1714-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6515123' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2021-11-19-1714-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-11-19-1714-pacific sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2021-11-19-1714-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa targets: smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAuSUreKceXGZnVQyUt+bRgFUwlqvehGX7blpu3zZTvxwZYBwM14Jsbgh4ViIngttBq9HCHlT5DZ/p9djhIdtTaBUPODRTGnpYjAqcrlB3Fvvc7lDaU5uYLRAXs4OpHanNQ2PhZPRSnasoIWQlThgnw5hkgqW4R/9glx8jjdhYE968082qxKT5lIqWgTKqLgY+bSrQAeSSMJoo/z1bdqoSEtKyN236ZLGGTBhr5nJhbQJQHePpuCxyK2i2nI7JBGB6/7lvxjh/u5JuK/nxFV3wuy6/B3WvNo8im8PFFW4argPMNAGx7DYfTwkd3jfz58B7wpC3EgM71NGKxx0k2FEfLLa//VBiTKHysI30IeecnZYoq/N2s7qluzBf4DQOzbHpj5gU2cRSdiQOqLfOsaEiABQMMpJAnxotViCLwAl8BlVNJMNc2CJCBaL6EyhrbXnV4lqWM7BdgRhhkvddPnBcz4PLpoF/6eJJVsnHfLM66KNo2wCRV3cORjlF5U7QVf0= smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4nK2O4jPWAYMOQ34V4vGMOsyqtrbOUp+GCTj9BUNl6woCD6MUF0o1wSF7UeK3j/r+21Iz/kU7UkiKi1MJJ5uV6ThTtGefFDcV5g7gNRx15DxQoD1UIfSRwgKZKXrx6VVKkcAV9Pk1Ll3Hlt/WY9WNjjSy0uEdjwUEP3owyu/VZmL87lxSx5m/Md8rAlp7uWRi+Ev4f9cVyrNW6x/MioZvFpZPU9/u3muivKawWizLmYavaVC4FxZ9mbjmZ3dY2RYvUwZ3k9bhnmu/R/TuanSf0YMDv6na5UkoniDm6eLJdw/3wQocik/FjLYh9Wbxl5ZeZ4/VUh8WuHeOeeL1eM/rDqm4H4IzcJE3pD4SIXplM9avOUyvD3R580yfYMUphH7dPmmmbbtZSXNiUh/euzG6GnBrX0R3/pYmwp4KpXiOrMvHly+7VB5tStsG2TBrALJks5oA34xHrlgxocFFrsqrKZdO8ep5J39vHjXB31c1MK91bXl1pqmYiTrT08ljMMk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v16.2.4 - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done kclient client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 6cfddd5e1c7d5ede1367cf3087c21af44eb88356 timestamp: 2021-11-20_04:32:48 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2021-11-20T07:20:12.182 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa; will attempt to use it 2021-11-20T07:20:12.183 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks 2021-11-20T07:20:12.183 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-11-20T07:20:12.185 INFO:teuthology.task.internal:Checking packages... 2021-11-20T07:20:12.214 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa' 2021-11-20T07:20:12.215 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-11-20T07:20:12.215 INFO:teuthology.packaging:ref: None 2021-11-20T07:20:12.215 INFO:teuthology.packaging:tag: None 2021-11-20T07:20:12.216 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T07:20:12.216 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:20:12.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T07:20:12.443 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-633.g9a97ad1b 2021-11-20T07:20:12.444 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-11-20T07:20:12.455 INFO:teuthology.task.internal:no buildpackages task found 2021-11-20T07:20:12.455 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-11-20T07:20:12.486 INFO:teuthology.task.internal:Saving configuration 2021-11-20T07:20:12.507 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-11-20T07:20:12.544 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-20T07:20:12.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 07:14:44.571584', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDSDtaSFPxVLMU0KRrccXeX/LBTIis1FPlXilBEyAucLacCp7oqhpzcQVvAOQTbRUU4AyZIAhwSjCTmFbQTZtiY='} 2021-11-20T07:20:12.595 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 07:14:44.570182', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLFY65cDrdFFA+wt5jhvGKTYW2u28GdaFzsk+bDjohPPwz2pwqE1x64xkES2f+BX6XJlymBkx24KuebNJyQO/v4='} 2021-11-20T07:20:12.595 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-11-20T07:20:12.606 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-20T07:20:12.607 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-20T07:20:12.607 INFO:teuthology.run_tasks:Running task console_log... 2021-11-20T07:20:12.693 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe0e3507598>, signals=[15]) 2021-11-20T07:20:12.696 INFO:teuthology.run_tasks:Running task internal.connect... 2021-11-20T07:20:12.708 INFO:teuthology.task.internal:Opening connections... 2021-11-20T07:20:12.709 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:20:12.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:20:12.781 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2021-11-20T07:20:12.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:20:12.853 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-11-20T07:20:12.864 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2021-11-20T07:20:12.885 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2021-11-20T07:20:12.886 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2021-11-20T07:20:12.944 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2021-11-20T07:20:12.944 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8" 2021-11-20T07:20:12.944 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2021-11-20T07:20:12.945 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2021-11-20T07:20:12.945 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8" 2021-11-20T07:20:12.945 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2021-11-20T07:20:12.946 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T07:20:12.946 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2021-11-20T07:20:12.946 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T07:20:12.946 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2021-11-20T07:20:12.946 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T07:20:12.947 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T07:20:12.947 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T07:20:12.948 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2021-11-20T07:20:12.967 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2021-11-20T07:20:12.985 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2021-11-20T07:20:12.986 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2021-11-20T07:20:13.043 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2021-11-20T07:20:13.044 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8" 2021-11-20T07:20:13.044 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2021-11-20T07:20:13.044 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2021-11-20T07:20:13.045 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8" 2021-11-20T07:20:13.045 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2021-11-20T07:20:13.045 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T07:20:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2021-11-20T07:20:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T07:20:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2021-11-20T07:20:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T07:20:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T07:20:13.047 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T07:20:13.047 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2021-11-20T07:20:13.066 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-11-20T07:20:13.083 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-11-20T07:20:13.100 INFO:teuthology.task.internal:Checking for old test directory... 2021-11-20T07:20:13.101 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2021-11-20T07:20:13.103 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2021-11-20T07:20:13.121 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-11-20T07:20:13.132 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-11-20T07:20:13.132 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2021-11-20T07:20:13.164 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2021-11-20T07:20:13.186 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-11-20T07:20:13.247 INFO:teuthology.run_tasks:Running task kernel... 2021-11-20T07:20:13.267 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-11-20T07:20:13.268 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-11-20T07:20:13.268 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2021-11-20T07:20:13.269 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2021-11-20T07:20:13.285 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T07:20:13.286 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-301.1.el8.x86_64 2021-11-20T07:20:13.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2021-11-20T07:20:13.288 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T07:20:13.288 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-301.1.el8.x86_64 2021-11-20T07:20:13.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2021-11-20T07:20:16.064 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - AppStream 38 MB/s | 17 MB 00:00 2021-11-20T07:20:16.170 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - AppStream 31 MB/s | 17 MB 00:00 2021-11-20T07:20:19.364 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Base 42 MB/s | 15 MB 00:00 2021-11-20T07:20:19.387 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - Base 42 MB/s | 15 MB 00:00 2021-11-20T07:20:21.571 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Extras 217 kB/s | 16 kB 00:00 2021-11-20T07:20:21.612 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - Extras 219 kB/s | 16 kB 00:00 2021-11-20T07:20:22.581 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T07:20:22.625 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T07:20:24.904 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 5.1 MB/s | 24 kB 00:00 2021-11-20T07:20:25.009 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2021-11-20T07:20:26.904 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T07:20:26.904 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T07:20:26.904 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T07:20:27.009 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T07:20:27.009 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T07:20:27.010 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T07:20:27.214 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:20:27.222 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:20:27.222 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:20:27.222 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:20:27.223 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:20:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T07:20:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T07:20:27.223 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T07:20:27.223 INFO:teuthology.orchestra.run.smithi071.stdout:Removing: 2021-11-20T07:20:27.224 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T07:20:27.224 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T07:20:27.224 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T07:20:27.224 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:20:27.225 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:20:27.225 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:20:27.225 INFO:teuthology.orchestra.run.smithi071.stdout:Install 3 Packages 2021-11-20T07:20:27.225 INFO:teuthology.orchestra.run.smithi071.stdout:Remove 3 Packages 2021-11-20T07:20:27.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:20:27.226 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 74 M 2021-11-20T07:20:27.227 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:20:27.330 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:20:27.332 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:20:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:20:27.332 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:20:27.332 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:20:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T07:20:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T07:20:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T07:20:27.333 INFO:teuthology.orchestra.run.smithi112.stdout:Removing: 2021-11-20T07:20:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T07:20:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T07:20:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T07:20:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:20:27.334 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:20:27.335 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:20:27.335 INFO:teuthology.orchestra.run.smithi112.stdout:Install 3 Packages 2021-11-20T07:20:27.335 INFO:teuthology.orchestra.run.smithi112.stdout:Remove 3 Packages 2021-11-20T07:20:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:20:27.336 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 74 M 2021-11-20T07:20:27.336 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:20:27.355 INFO:teuthology.orchestra.run.smithi071.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 51 MB/s | 7.0 MB 00:00 2021-11-20T07:20:28.159 INFO:teuthology.orchestra.run.smithi112.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 8.5 MB/s | 7.0 MB 00:00 2021-11-20T07:20:28.347 INFO:teuthology.orchestra.run.smithi071.stdout:(2/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 33 MB/s | 38 MB 00:01 2021-11-20T07:20:28.660 INFO:teuthology.orchestra.run.smithi112.stdout:(2/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 22 MB/s | 30 MB 00:01 2021-11-20T07:20:28.671 INFO:teuthology.orchestra.run.smithi071.stdout:(3/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 21 MB/s | 30 MB 00:01 2021-11-20T07:20:28.675 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:20:28.676 INFO:teuthology.orchestra.run.smithi071.stdout:Total 51 MB/s | 74 MB 00:01 2021-11-20T07:20:28.808 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:20:29.314 INFO:teuthology.orchestra.run.smithi112.stdout:(3/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 19 MB/s | 38 MB 00:01 2021-11-20T07:20:29.318 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:20:29.319 INFO:teuthology.orchestra.run.smithi112.stdout:Total 37 MB/s | 74 MB 00:01 2021-11-20T07:20:29.451 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:20:29.469 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:20:29.470 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:20:30.126 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:20:30.127 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:20:30.558 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:20:30.578 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:20:31.217 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:20:31.267 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:20:33.262 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:20:33.808 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:20:33.951 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:20:34.479 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:20:38.014 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:20:38.399 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:20:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:20:39.128 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:20:41.461 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:20:41.653 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T07:20:41.653 INFO:teuthology.orchestra.run.smithi071.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:20:41.856 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:20:41.856 INFO:teuthology.orchestra.run.smithi071.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:20:42.193 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:20:42.365 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T07:20:42.365 INFO:teuthology.orchestra.run.smithi112.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:20:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:20:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:20:44.022 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:20:44.224 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:20:44.234 INFO:teuthology.orchestra.run.smithi071.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:20:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:20:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:20:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:20:47.133 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:20:47.804 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:21:07.756 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T07:21:08.121 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:21:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:21:08.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:21:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T07:21:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:21:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:21:08.399 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T07:21:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:21:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:21:08.614 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:21:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:08.615 INFO:teuthology.orchestra.run.smithi071.stdout:Removed: 2021-11-20T07:21:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T07:21:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T07:21:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:21:08.616 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:21:08.723 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel --last | head -n 1 2021-11-20T07:21:08.807 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:21:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T07:21:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T07:21:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T07:21:08.808 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T07:21:08.809 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T07:21:08.921 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 07:20:41 AM UTC 2021-11-20T07:21:08.923 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:08.923 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:08.924 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T07:21:08.924 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T07:21:08.924 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2021-11-20T07:21:08.924 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T07:21:08.925 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel --last | head -n 1 2021-11-20T07:21:09.067 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 07:20:41 AM UTC 2021-11-20T07:21:09.069 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2021-11-20T07:21:09.100 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2021-11-20T07:21:09.101 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2021-11-20T07:21:09.101 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2021-11-20T07:21:09.101 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 99.el8 2021-11-20T07:21:09.102 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2021-11-20T07:21:09.102 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T07:21:09.102 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2021-11-20T07:21:09.102 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9264704 2021-11-20T07:21:09.103 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2021-11-20T07:21:09.103 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T07:21:09.103 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T07:21:09.103 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T07:21:09.103 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T07:21:09.104 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2021-11-20T07:21:09.104 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : CentOS BuildSystem 2021-11-20T07:21:09.105 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2021-11-20T07:21:09.105 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T07:21:09.105 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2021-11-20T07:21:09.105 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2021-11-20T07:21:09.105 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:21:09.106 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T07:21:09.106 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T07:21:09.106 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T07:21:09.108 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2021-11-20T07:21:09.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:21:09.109 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T07:21:09.110 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T07:21:09.111 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T07:21:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T07:21:09.264 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:21:09.264 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:21:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.265 INFO:teuthology.orchestra.run.smithi112.stdout:Removed: 2021-11-20T07:21:09.265 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T07:21:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T07:21:09.266 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:21:09.266 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:21:09.387 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel --last | head -n 1 2021-11-20T07:21:09.548 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 07:20:42 AM UTC 2021-11-20T07:21:09.550 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.550 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.550 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T07:21:09.551 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T07:21:09.551 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2021-11-20T07:21:09.551 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T07:21:09.552 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel --last | head -n 1 2021-11-20T07:21:09.693 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 07:20:42 AM UTC 2021-11-20T07:21:09.695 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2021-11-20T07:21:09.726 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2021-11-20T07:21:09.726 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2021-11-20T07:21:09.727 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.02 2021-11-20T07:21:09.727 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 99.el8 2021-11-20T07:21:09.727 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2021-11-20T07:21:09.727 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T07:21:09.728 INFO:teuthology.orchestra.run.smithi112.stdout:Group : System Environment/Base 2021-11-20T07:21:09.728 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 9264704 2021-11-20T07:21:09.728 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2021-11-20T07:21:09.728 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T07:21:09.729 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T07:21:09.729 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T07:21:09.729 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T07:21:09.729 INFO:teuthology.orchestra.run.smithi112.stdout:Relocations : (not relocatable) 2021-11-20T07:21:09.729 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : CentOS BuildSystem 2021-11-20T07:21:09.730 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2021-11-20T07:21:09.730 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T07:21:09.730 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2021-11-20T07:21:09.730 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2021-11-20T07:21:09.731 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:21:09.731 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T07:21:09.731 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T07:21:09.731 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T07:21:09.731 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2021-11-20T07:21:09.732 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:21:09.733 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T07:21:09.734 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T07:21:09.734 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:09.738 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T07:21:10.848 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2021-11-20T07:21:11.107 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2021-11-20T07:21:12.699 INFO:teuthology.orchestra.run.smithi071.stderr:done 2021-11-20T07:21:12.701 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2021-11-20T07:21:12.732 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.AgAXH26d9Y 2021-11-20T07:21:12.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AgAXH26d9Y 2021-11-20T07:21:12.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.AgAXH26d9Y 2021-11-20T07:21:12.949 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.AgAXH26d9Y is 6KB 2021-11-20T07:21:13.008 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.AgAXH26d9Y 2021-11-20T07:21:13.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T07:21:13.103 INFO:teuthology.orchestra.run.smithi071.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T07:21:13.103 INFO:teuthology.orchestra.run.smithi071.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T07:21:13.103 INFO:teuthology.orchestra.run.smithi071.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T07:21:13.104 INFO:teuthology.orchestra.run.smithi071.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T07:21:13.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:13.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2021-11-20T07:21:13.725 INFO:teuthology.orchestra.run.smithi112.stderr:done 2021-11-20T07:21:13.728 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2021-11-20T07:21:13.748 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.M9PWOGTPrx 2021-11-20T07:21:13.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.M9PWOGTPrx 2021-11-20T07:21:13.820 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.M9PWOGTPrx 2021-11-20T07:21:13.959 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.M9PWOGTPrx is 6KB 2021-11-20T07:21:14.011 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.M9PWOGTPrx 2021-11-20T07:21:14.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T07:21:14.096 INFO:teuthology.orchestra.run.smithi112.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T07:21:14.097 INFO:teuthology.orchestra.run.smithi112.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T07:21:14.097 INFO:teuthology.orchestra.run.smithi112.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T07:21:14.097 INFO:teuthology.orchestra.run.smithi112.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T07:21:14.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:21:14.277 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2021-11-20T07:21:43.283 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2021-11-20T07:21:43.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2021-11-20T07:21:43.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:21:44.282 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2021-11-20T07:21:44.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2021-11-20T07:21:44.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:23:01.863 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T07:23:02.832 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T07:23:31.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2021-11-20T07:23:31.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:23:32.284 DEBUG:teuthology.orchestra.run.smithi071:> true 2021-11-20T07:23:32.548 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2021-11-20T07:23:32.549 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T07:23:32.549 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2021-11-20T07:23:32.608 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:23:32.609 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:23:32.609 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T07:23:32.609 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T07:23:32.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2021-11-20T07:23:32.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:23:33.278 DEBUG:teuthology.orchestra.run.smithi112:> true 2021-11-20T07:23:33.567 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2021-11-20T07:23:33.568 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T07:23:33.568 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2021-11-20T07:23:33.584 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:23:33.585 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T07:23:33.585 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T07:23:33.585 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T07:23:33.610 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2021-11-20T07:23:33.610 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T07:23:33.697 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2021-11-20T07:23:33.717 DEBUG:teuthology.parallel:result is None 2021-11-20T07:23:34.586 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2021-11-20T07:23:34.587 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T07:23:34.705 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2021-11-20T07:23:34.721 DEBUG:teuthology.parallel:result is None 2021-11-20T07:23:34.722 INFO:teuthology.run_tasks:Running task internal.base... 2021-11-20T07:23:34.734 INFO:teuthology.task.internal:Creating test directory... 2021-11-20T07:23:34.734 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T07:23:34.737 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T07:23:34.756 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-11-20T07:23:34.767 INFO:teuthology.run_tasks:Running task internal.archive... 2021-11-20T07:23:34.785 INFO:teuthology.task.internal:Creating archive directory... 2021-11-20T07:23:34.785 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T07:23:34.796 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T07:23:34.867 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-11-20T07:23:34.875 INFO:teuthology.task.internal:Enabling coredump saving... 2021-11-20T07:23:34.876 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T07:23:34.878 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T07:23:34.920 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T07:23:34.925 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T07:23:34.932 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T07:23:34.937 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T07:23:34.938 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-11-20T07:23:34.947 INFO:teuthology.task.internal:Configuring sudo... 2021-11-20T07:23:34.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T07:23:34.975 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T07:23:35.003 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-11-20T07:23:35.015 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-11-20T07:23:35.016 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T07:23:35.044 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T07:23:35.060 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T07:23:35.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T07:23:35.199 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T07:23:35.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T07:23:35.325 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:23:35.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T07:23:35.393 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T07:23:35.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T07:23:35.506 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T07:23:35.555 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T07:23:35.594 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:23:35.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T07:23:35.661 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2021-11-20T07:23:35.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2021-11-20T07:23:35.730 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T07:23:35.756 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T07:23:36.081 INFO:teuthology.run_tasks:Running task internal.timer... 2021-11-20T07:23:36.092 INFO:teuthology.task.internal:Starting timer... 2021-11-20T07:23:36.093 INFO:teuthology.run_tasks:Running task pcp... 2021-11-20T07:23:36.132 INFO:teuthology.run_tasks:Running task selinux... 2021-11-20T07:23:36.151 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-11-20T07:23:36.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2021-11-20T07:23:36.214 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: [ OK ] 2021-11-20T07:23:36.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2021-11-20T07:23:36.289 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: [ OK ] 2021-11-20T07:23:36.291 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-11-20T07:23:36.291 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2021-11-20T07:23:36.318 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2021-11-20T07:23:36.319 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2021-11-20T07:23:36.346 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2021-11-20T07:23:36.347 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2021-11-20T07:23:36.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T07:23:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:23:36.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T07:23:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:23:36.414 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-11-20T07:23:36.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2021-11-20T07:23:36.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2021-11-20T07:23:36.488 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-11-20T07:23:36.506 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-11-20T07:23:36.515 INFO:teuthology.repo_utils:Fetching master from origin 2021-11-20T07:23:36.551 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-11-20T07:23:36.572 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-11-20T07:23:36.573 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-11-20T07:23:37.009 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-11-20T07:23:37.830 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-11-20T07:23:40.952 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-11-20T07:23:41.057 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:41 +0000 (0:00:00.125) 0:00:00.125 ***** 2021-11-20T07:23:41.068 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:41 +0000 (0:00:00.011) 0:00:00.136 ***** 2021-11-20T07:23:42.518 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-11-20T07:23:42.567 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:42 +0000 (0:00:01.498) 0:00:01.635 ***** 2021-11-20T07:23:42.837 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-20T07:23:42.839 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T07:23:42.860 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:42 +0000 (0:00:00.291) 0:00:01.927 ***** 2021-11-20T07:23:43.401 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-11-20T07:23:43.403 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T07:23:43.424 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:43 +0000 (0:00:00.564) 0:00:02.492 ***** 2021-11-20T07:23:43.493 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-11-20T07:23:43.495 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:43.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:43 +0000 (0:00:00.090) 0:00:02.582 ***** 2021-11-20T07:23:43.545 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-20T07:23:43.547 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2021-11-20T07:23:43.593 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:43 +0000 (0:00:00.078) 0:00:02.661 ***** 2021-11-20T07:23:43.773 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-20T07:23:43.774 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:23:43.795 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:43 +0000 (0:00:00.201) 0:00:02.863 ***** 2021-11-20T07:23:43.801 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-20T07:23:43.803 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T07:23:43.825 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:43 +0000 (0:00:00.029) 0:00:02.893 ***** 2021-11-20T07:23:44.125 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi112.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T07:23:44.147 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:44 +0000 (0:00:00.320) 0:00:03.214 ***** 2021-11-20T07:23:44.214 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:44.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:44 +0000 (0:00:00.091) 0:00:03.305 ***** 2021-11-20T07:23:44.420 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-20T07:23:44.421 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:23:44.442 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:44 +0000 (0:00:00.205) 0:00:03.510 ***** 2021-11-20T07:23:44.720 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393023.8412614-2509-113217697566064/source", "state": "file", "uid": 0} 2021-11-20T07:23:44.744 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:44 +0000 (0:00:00.302) 0:00:03.813 ***** 2021-11-20T07:23:45.077 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393024.4896722-2618-130899149365254/source", "state": "file", "uid": 0} 2021-11-20T07:23:45.098 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:45 +0000 (0:00:00.353) 0:00:04.166 ***** 2021-11-20T07:23:45.168 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-11-20T07:23:45.170 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T07:23:45.172 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T07:23:45.418 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T07:23:45.420 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T07:23:45.484 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-11-20T07:23:45.485 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T07:23:45.499 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 20 November 2021 07:23:45 +0000 (0:00:00.401) 0:00:04.567 ***** 2021-11-20T07:23:46.014 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] 2021-11-20T07:23:47.061 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2021-11-20T07:23:47.079 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Saturday 20 November 2021 07:23:47 +0000 (0:00:01.579) 0:00:06.147 ***** 2021-11-20T07:23:47.103 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T07:23:47.114 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T07:23:47.119 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T07:23:47.270 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:47 +0000 (0:00:00.190) 0:00:06.337 ***** 2021-11-20T07:23:47.290 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:47 +0000 (0:00:00.020) 0:00:06.358 ***** 2021-11-20T07:23:48.932 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-11-20T07:23:48.974 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2021-11-20T07:23:49.007 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Saturday 20 November 2021 07:23:49 +0000 (0:00:01.714) 0:00:08.072 ***** 2021-11-20T07:23:49.058 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:49 +0000 (0:00:00.053) 0:00:08.125 ***** 2021-11-20T07:23:49.714 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-11-20T07:23:49.716 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.189665", "end": "2021-11-20 07:23:49.690453", "rc": 0, "start": "2021-11-20 07:23:49.500788", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T07:23:49.717 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.185051", "end": "2021-11-20 07:23:49.686882", "rc": 0, "start": "2021-11-20 07:23:49.501831", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T07:23:49.742 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:49 +0000 (0:00:00.684) 0:00:08.810 ***** 2021-11-20T07:23:49.763 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:49 +0000 (0:00:00.020) 0:00:08.831 ***** 2021-11-20T07:23:49.960 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-11-20T07:23:49.962 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012949", "end": "2021-11-20 07:23:49.937551", "failed_when_result": false, "rc": 0, "start": "2021-11-20 07:23:49.924602", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T07:23:49.982 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:49 +0000 (0:00:00.219) 0:00:09.050 ***** 2021-11-20T07:23:49.990 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.006484", "end": "2021-11-20 07:23:49.943432", "failed_when_result": false, "rc": 0, "start": "2021-11-20 07:23:49.936948", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T07:23:50.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.031) 0:00:09.082 ***** 2021-11-20T07:23:50.055 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.078 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.064) 0:00:09.146 ***** 2021-11-20T07:23:50.097 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.122 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.043) 0:00:09.190 ***** 2021-11-20T07:23:50.156 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.178 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.055) 0:00:09.246 ***** 2021-11-20T07:23:50.197 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.222 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.043) 0:00:09.290 ***** 2021-11-20T07:23:50.251 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-11-20T07:23:50.255 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.276 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.053) 0:00:09.343 ***** 2021-11-20T07:23:50.294 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.320 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.042) 0:00:09.386 ***** 2021-11-20T07:23:50.358 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.385 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.067) 0:00:09.453 ***** 2021-11-20T07:23:50.392 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.420 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.032) 0:00:09.486 ***** 2021-11-20T07:23:50.457 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-11-20T07:23:50.459 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.063) 0:00:09.549 ***** 2021-11-20T07:23:50.499 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:23:50.524 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.042) 0:00:09.592 ***** 2021-11-20T07:23:50.713 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-20T07:23:50.715 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:23:50.738 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.214) 0:00:09.806 ***** 2021-11-20T07:23:50.756 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:23:50.781 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:23:50 +0000 (0:00:00.042) 0:00:09.849 ***** 2021-11-20T07:23:51.016 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T07:23:51.018 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T07:23:51.070 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T07:23:51.218 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T07:23:51.271 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T07:23:51.420 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T07:23:51.474 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T07:23:51.611 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T07:23:51.664 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T07:23:51.815 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T07:23:51.858 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T07:23:52.006 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T07:23:52.059 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T07:23:52.206 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T07:23:52.259 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T07:23:52.396 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T07:23:52.459 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T07:23:52.597 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T07:23:52.650 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T07:23:52.790 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T07:23:52.854 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T07:23:52.991 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T07:23:53.058 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T07:23:53.182 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T07:23:53.246 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T07:23:53.382 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T07:23:53.436 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T07:23:53.575 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T07:23:53.642 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T07:23:53.771 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T07:23:53.835 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T07:23:53.974 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T07:23:54.032 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T07:23:54.187 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T07:23:54.248 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T07:23:54.375 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T07:23:54.440 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T07:23:54.567 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T07:23:54.631 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T07:23:54.763 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T07:23:54.830 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T07:23:54.968 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T07:23:55.031 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T07:23:55.167 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T07:23:55.230 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T07:23:55.366 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T07:23:55.419 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T07:23:55.568 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T07:23:55.623 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T07:23:55.763 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T07:23:55.816 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T07:23:55.953 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T07:23:56.020 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T07:23:56.154 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T07:23:56.207 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T07:23:56.355 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T07:23:56.409 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T07:23:56.547 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T07:23:56.600 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T07:23:56.738 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T07:23:56.802 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T07:23:56.939 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T07:23:56.992 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T07:23:57.129 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T07:23:57.193 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T07:23:57.329 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T07:23:57.393 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T07:23:57.520 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T07:23:57.595 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T07:23:57.951 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T07:23:57.977 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T07:23:58.009 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T07:23:58.173 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T07:23:58.212 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T07:23:58.372 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T07:23:58.407 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T07:23:58.589 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T07:23:58.600 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T07:23:58.791 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T07:23:58.802 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T07:23:58.998 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T07:23:59.200 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T07:23:59.212 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T07:23:59.401 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T07:23:59.401 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T07:23:59.604 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T07:23:59.799 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T07:23:59.801 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T07:23:59.990 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T07:23:59.991 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T07:24:00.190 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T07:24:00.191 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T07:24:00.390 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T07:24:00.401 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T07:24:00.579 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T07:24:00.594 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T07:24:00.774 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T07:24:00.797 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T07:24:00.975 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T07:24:00.997 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T07:24:01.174 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T07:24:01.196 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T07:24:01.375 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T07:24:01.386 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T07:24:01.564 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T07:24:01.586 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T07:24:01.765 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T07:24:01.789 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T07:24:01.958 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T07:24:01.980 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T07:24:02.161 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T07:24:02.184 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T07:24:02.362 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T07:24:02.384 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T07:24:02.563 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T07:24:02.574 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T07:24:02.746 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T07:24:02.779 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T07:24:02.957 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T07:24:03.014 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T07:24:03.153 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T07:24:03.219 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T07:24:03.367 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T07:24:03.420 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T07:24:03.558 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T07:24:03.623 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T07:24:03.763 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T07:24:03.831 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T07:24:03.955 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T07:24:04.030 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T07:24:04.155 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T07:24:04.228 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T07:24:04.355 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T07:24:04.419 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T07:24:04.554 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T07:24:04.618 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T07:24:04.765 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T07:24:04.818 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T07:24:04.956 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T07:24:05.020 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T07:24:05.146 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T07:24:05.210 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T07:24:05.357 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T07:24:05.423 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T07:24:05.561 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T07:24:05.613 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T07:24:05.759 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T07:24:05.815 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T07:24:05.962 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T07:24:06.015 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T07:24:06.163 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T07:24:06.215 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T07:24:06.351 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T07:24:06.415 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T07:24:06.554 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T07:24:06.617 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T07:24:06.755 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T07:24:06.809 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T07:24:06.945 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T07:24:07.009 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T07:24:07.145 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T07:24:07.209 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T07:24:07.335 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T07:24:07.410 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T07:24:07.538 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T07:24:07.603 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T07:24:07.741 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T07:24:07.804 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T07:24:07.937 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T07:24:08.011 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T07:24:08.137 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T07:24:08.212 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T07:24:08.337 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T07:24:08.411 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T07:24:08.526 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T07:24:08.611 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T07:24:08.726 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T07:24:08.811 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T07:24:08.928 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T07:24:09.003 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T07:24:09.127 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T07:24:09.200 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T07:24:09.327 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T07:24:09.402 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T07:24:09.520 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T07:24:09.594 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T07:24:09.721 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T07:24:09.795 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T07:24:09.913 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T07:24:09.989 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T07:24:10.125 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T07:24:10.179 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T07:24:10.316 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T07:24:10.381 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T07:24:10.517 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T07:24:10.583 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T07:24:10.709 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T07:24:10.773 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T07:24:10.911 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T07:24:10.974 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T07:24:11.102 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T07:24:11.182 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T07:24:11.183 INFO:teuthology.task.ansible.out: 2021-11-20T07:24:11.300 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T07:24:11.375 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T07:24:11.503 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T07:24:11.567 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T07:24:11.705 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T07:24:11.779 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T07:24:11.916 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T07:24:11.975 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T07:24:12.114 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T07:24:12.179 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T07:24:12.317 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T07:24:12.392 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T07:24:12.514 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T07:24:12.578 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T07:24:12.709 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T07:24:12.784 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T07:24:12.911 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T07:24:12.976 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T07:24:13.124 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T07:24:13.188 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T07:24:13.307 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T07:24:13.378 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T07:24:13.506 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T07:24:13.571 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T07:24:13.703 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T07:24:13.781 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T07:24:13.896 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T07:24:13.973 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T07:24:14.097 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T07:24:14.171 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T07:24:14.301 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T07:24:14.376 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T07:24:14.494 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T07:24:14.578 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T07:24:14.698 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T07:24:14.778 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T07:24:14.896 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T07:24:14.970 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T07:24:15.087 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T07:24:15.173 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T07:24:15.287 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T07:24:15.371 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T07:24:15.487 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T07:24:15.561 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T07:24:15.689 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T07:24:15.764 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T07:24:15.892 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T07:24:15.969 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T07:24:16.098 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T07:24:16.162 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T07:24:16.288 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T07:24:16.352 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T07:24:16.490 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T07:24:16.554 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T07:24:16.682 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T07:24:16.757 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T07:24:16.876 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T07:24:16.966 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T07:24:17.083 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T07:24:17.170 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T07:24:17.287 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T07:24:17.360 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T07:24:18.826 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T07:24:18.962 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T07:24:20.638 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T07:24:20.880 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T07:24:22.191 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T07:24:22.403 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T07:24:23.705 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T07:24:23.955 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T07:24:25.233 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T07:24:25.479 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T07:24:26.781 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T07:24:27.015 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T07:24:28.316 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T07:24:28.558 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T07:24:29.941 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T07:24:30.170 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T07:24:31.574 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T07:24:31.796 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T07:24:33.220 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T07:24:33.421 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T07:24:34.884 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T07:24:35.042 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T07:24:36.553 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T07:24:36.670 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T07:24:38.206 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T07:24:38.355 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T07:24:39.870 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T07:24:39.965 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T07:24:41.583 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T07:24:41.605 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T07:24:43.214 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T07:24:43.311 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T07:24:44.838 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T07:24:44.956 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T07:24:46.529 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T07:24:46.656 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T07:24:46.658 INFO:teuthology.task.ansible.out: 2021-11-20T07:24:48.202 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T07:24:48.369 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T07:24:49.873 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T07:24:49.978 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T07:24:51.600 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T07:24:51.738 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T07:24:51.760 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:51 +0000 (0:01:00.979) 0:01:10.828 ***** 2021-11-20T07:24:51.855 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T07:24:51.885 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:51 +0000 (0:00:00.124) 0:01:10.953 ***** 2021-11-20T07:24:51.915 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:51 +0000 (0:00:00.029) 0:01:10.983 ***** 2021-11-20T07:24:51.972 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:51 +0000 (0:00:00.055) 0:01:11.038 ***** 2021-11-20T07:24:52.154 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c 2021-11-20T07:24:52.157 INFO:teuthology.task.ansible.out:68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": " 2021-11-20T07:24:52.158 INFO:teuthology.task.ansible.out:ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\ 2021-11-20T07:24:52.160 INFO:teuthology.task.ansible.out:nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD 2021-11-20T07:24:52.161 INFO:teuthology.task.ansible.out:2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu" 2021-11-20T07:24:52.162 INFO:teuthology.task.ansible.out:, "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f 2021-11-20T07:24:52.163 INFO:teuthology.task.ansible.out:11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73 2021-11-20T07:24:52.166 INFO:teuthology.task.ansible.out:d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259 2021-11-20T07:24:52.167 INFO:teuthology.task.ansible.out:b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5 2021-11-20T07:24:52.170 INFO:teuthology.task.ansible.out:a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, 2021-11-20T07:24:52.180 INFO:teuthology.task.ansible.out: {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271 2021-11-20T07:24:52.182 INFO:teuthology.task.ansible.out:f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f 2021-11-20T07:24:52.184 INFO:teuthology.task.ansible.out:41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59c 2021-11-20T07:24:52.188 INFO:teuthology.task.ansible.out:bf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe 2021-11-20T07:24:52.193 INFO:teuthology.task.ansible.out:802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T07:24:52.218 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:52 +0000 (0:00:00.247) 0:01:11.285 ***** 2021-11-20T07:24:52.230 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T07:24:52.234 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T07:24:52.236 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T07:24:52.238 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T07:24:52.240 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T07:24:52.245 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T07:24:52.247 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T07:24:52.251 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T07:24:52.254 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T07:24:52.256 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T07:24:52.257 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T07:24:52.260 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T07:24:52.262 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-11-20T07:24:52.263 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T07:24:52.265 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T07:24:52.291 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:52 +0000 (0:00:00.073) 0:01:11.359 ***** 2021-11-20T07:24:53.127 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T07:24:53.129 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T07:24:53.150 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:53 +0000 (0:00:00.858) 0:01:12.217 ***** 2021-11-20T07:24:53.169 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T07:24:53.192 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:24:53 +0000 (0:00:00.042) 0:01:12.260 ***** 2021-11-20T07:24:53.496 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T07:24:53.498 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T07:24:53.549 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T07:24:53.708 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T07:24:53.751 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T07:24:53.921 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T07:24:53.976 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T07:24:54.125 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T07:24:54.127 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T07:24:54.204 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T07:24:54.207 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T07:24:54.334 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T07:24:54.412 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T07:24:54.711 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T07:24:54.712 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T07:24:54.809 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T07:24:54.816 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T07:24:54.923 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T07:24:55.018 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T07:24:55.125 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T07:24:55.230 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T07:24:55.336 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T07:24:55.348 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:55.442 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T07:24:55.453 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:55.547 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T07:24:55.549 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T07:24:55.674 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T07:24:55.677 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T07:24:55.768 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T07:24:55.771 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T07:24:55.780 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:55.791 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:55.886 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T07:24:55.888 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T07:24:55.898 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:55.900 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:24:56.005 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T07:24:56.110 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T07:24:56.220 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T07:24:56.222 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T07:24:56.315 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T07:24:56.317 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T07:24:56.439 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T07:24:56.528 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T07:24:56.656 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T07:24:56.732 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T07:24:56.878 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T07:24:56.963 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T07:24:57.105 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T07:24:57.107 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T07:24:57.170 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T07:24:57.172 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T07:24:57.307 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T07:24:57.308 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T07:24:57.384 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T07:24:57.387 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T07:24:57.558 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T07:24:57.589 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T07:24:57.726 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T07:24:57.790 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T07:24:57.937 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T07:24:57.939 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T07:24:58.011 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T07:24:58.013 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T07:24:58.161 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T07:24:58.214 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T07:24:58.364 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T07:24:58.429 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T07:24:58.575 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T07:24:58.629 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T07:24:58.777 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T07:24:58.851 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T07:24:58.989 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T07:24:59.053 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T07:24:59.202 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T07:24:59.203 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T07:24:59.266 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T07:24:59.268 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T07:24:59.412 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T07:24:59.476 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T07:24:59.614 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T07:24:59.678 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T07:24:59.832 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T07:24:59.906 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T07:25:00.079 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T07:25:00.123 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T07:25:00.281 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T07:25:00.334 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T07:25:00.491 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T07:25:00.533 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T07:25:00.701 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T07:25:00.744 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T07:25:00.914 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T07:25:00.916 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T07:25:00.946 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T07:25:00.948 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T07:25:01.123 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T07:25:01.125 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T07:25:01.156 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T07:25:01.158 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T07:25:01.335 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T07:25:01.367 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T07:25:01.546 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T07:25:01.578 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T07:25:01.746 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T07:25:01.788 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T07:25:01.955 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T07:25:01.958 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T07:25:01.987 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T07:25:01.989 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T07:25:02.178 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T07:25:02.180 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T07:25:02.182 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T07:25:02.200 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T07:25:02.202 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T07:25:02.204 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T07:25:02.389 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T07:25:02.424 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T07:25:02.593 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T07:25:02.646 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T07:25:02.805 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T07:25:02.858 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T07:25:03.004 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T07:25:03.067 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T07:25:03.214 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T07:25:03.278 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T07:25:03.415 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T07:25:03.417 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T07:25:03.489 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T07:25:03.490 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T07:25:03.627 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T07:25:03.732 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T07:25:03.852 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T07:25:03.918 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T07:25:04.066 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T07:25:04.068 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T07:25:04.129 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T07:25:04.131 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T07:25:04.265 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T07:25:04.329 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T07:25:04.476 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T07:25:04.551 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T07:25:04.687 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T07:25:04.689 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T07:25:04.750 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T07:25:04.752 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T07:25:04.890 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T07:25:04.964 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T07:25:05.099 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T07:25:05.174 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T07:25:05.303 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T07:25:05.388 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T07:25:05.514 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T07:25:05.589 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T07:25:05.726 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T07:25:05.800 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T07:25:05.937 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T07:25:06.000 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T07:25:06.147 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T07:25:06.234 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T07:25:06.372 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T07:25:06.446 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T07:25:06.584 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T07:25:06.657 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T07:25:06.805 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T07:25:06.868 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T07:25:07.015 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T07:25:07.068 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T07:25:07.241 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T07:25:07.284 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T07:25:07.442 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T07:25:07.496 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T07:25:07.653 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T07:25:07.706 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T07:25:07.862 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T07:25:07.915 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T07:25:08.074 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T07:25:08.127 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T07:25:08.275 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T07:25:08.328 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T07:25:08.485 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T07:25:08.538 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T07:25:08.695 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T07:25:08.748 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T07:25:08.908 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T07:25:08.995 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T07:25:09.160 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T07:25:09.162 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T07:25:09.234 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T07:25:09.236 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T07:25:09.384 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T07:25:09.458 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T07:25:09.599 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T07:25:09.675 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T07:25:09.812 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T07:25:09.886 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T07:25:10.014 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T07:25:10.016 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T07:25:10.098 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T07:25:10.100 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T07:25:10.224 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T07:25:10.298 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T07:25:10.429 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T07:25:10.513 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T07:25:10.649 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T07:25:10.724 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T07:25:10.849 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T07:25:10.933 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T07:25:11.059 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T07:25:11.143 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T07:25:11.269 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T07:25:11.343 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T07:25:11.470 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T07:25:11.557 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T07:25:11.693 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T07:25:11.695 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T07:25:11.779 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T07:25:11.781 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T07:25:11.895 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T07:25:11.990 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T07:25:12.095 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T07:25:12.201 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T07:25:12.305 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T07:25:12.402 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T07:25:12.519 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T07:25:12.623 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T07:25:12.729 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T07:25:12.834 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T07:25:12.929 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T07:25:13.034 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T07:25:13.139 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T07:25:13.141 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T07:25:13.244 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T07:25:13.245 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T07:25:13.349 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T07:25:13.455 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T07:25:13.560 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T07:25:13.666 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T07:25:13.761 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T07:25:13.882 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T07:25:13.968 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T07:25:14.096 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T07:25:14.181 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T07:25:14.306 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T07:25:14.393 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T07:25:14.505 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T07:25:14.600 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T07:25:14.709 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T07:25:14.803 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T07:25:14.922 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T07:25:15.027 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T07:25:15.132 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T07:25:15.228 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T07:25:15.343 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T07:25:15.438 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T07:25:15.554 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T07:25:15.639 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T07:25:15.756 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T07:25:15.852 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T07:25:15.958 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T07:25:16.064 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T07:25:16.173 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T07:25:16.278 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T07:25:16.385 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T07:25:16.491 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T07:25:16.596 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T07:25:16.692 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T07:25:16.797 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T07:25:16.902 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T07:25:17.008 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T07:25:17.112 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T07:25:17.114 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T07:25:17.227 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T07:25:17.229 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T07:25:17.321 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T07:25:17.436 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T07:25:17.532 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T07:25:17.636 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T07:25:17.742 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T07:25:17.848 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T07:25:17.944 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T07:25:18.049 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T07:25:18.157 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T07:25:18.262 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T07:25:18.368 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T07:25:18.476 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T07:25:18.593 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T07:25:18.688 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T07:25:18.814 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T07:25:18.908 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T07:25:19.024 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T07:25:19.108 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T07:25:19.234 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T07:25:19.308 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T07:25:19.445 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T07:25:19.510 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T07:25:19.646 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T07:25:19.731 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T07:25:19.867 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T07:25:19.869 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T07:25:19.942 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T07:25:19.944 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T07:25:20.078 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T07:25:20.153 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T07:25:20.282 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T07:25:20.367 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T07:25:20.493 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T07:25:20.499 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T07:25:20.577 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T07:25:20.579 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T07:25:20.707 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T07:25:20.789 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T07:25:20.906 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T07:25:21.002 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T07:25:21.118 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T07:25:21.204 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T07:25:21.321 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T07:25:21.428 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T07:25:21.533 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T07:25:21.638 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T07:25:21.742 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T07:25:21.850 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T07:25:21.954 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T07:25:22.059 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T07:25:22.166 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T07:25:22.274 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T07:25:22.369 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T07:25:22.498 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T07:25:22.581 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T07:25:22.708 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T07:25:22.804 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T07:25:22.910 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T07:25:23.005 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T07:25:23.123 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T07:25:23.221 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T07:25:23.326 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T07:25:23.433 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T07:25:23.543 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T07:25:23.640 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T07:25:23.758 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T07:25:23.863 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T07:25:23.970 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T07:25:24.075 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T07:25:24.172 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T07:25:24.276 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T07:25:24.384 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T07:25:24.508 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T07:25:24.605 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T07:25:24.722 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T07:25:24.816 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T07:25:24.922 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T07:25:25.018 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T07:25:25.136 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T07:25:25.232 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T07:25:25.348 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T07:25:25.445 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:25 +0000 (0:00:32.252) 0:01:44.513 ***** 2021-11-20T07:25:25.455 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T07:25:25.551 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:25 +0000 (0:00:00.105) 0:01:44.618 ***** 2021-11-20T07:25:25.560 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T07:25:25.561 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.562 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ 2021-11-20T07:25:25.564 INFO:teuthology.task.ansible.out:ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.577 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.579 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.579 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.591 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.643 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.646 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.655 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.657 INFO:teuthology.task.ansible.out: 2021-11-20T07:25:25.667 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.669 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.678 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.689 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.691 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.701 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.807 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T07:25:25.818 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.821 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.924 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T07:25:25.926 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:25.936 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.032 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T07:25:26.158 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T07:25:26.242 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T07:25:26.254 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.255 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.265 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.266 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.276 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.287 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.289 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.298 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.300 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.310 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.321 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.323 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.332 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.343 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.345 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.354 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.356 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f968 2021-11-20T07:25:26.358 INFO:teuthology.task.ansible.out:5dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.366 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.368 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.368 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.370 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.380 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.382 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.393 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.404 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.405 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0b 2021-11-20T07:25:26.408 INFO:teuthology.task.ansible.out:b0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.416 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.417 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.428 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.430 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.440 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.442 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.452 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.453 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.454 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.465 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.466 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.466 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.477 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.479 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.488 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.490 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.500 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.501 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.513 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.514 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.526 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.527 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.539 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.541 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.550 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.552 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.562 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.564 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.574 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.575 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.587 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.589 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.600 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.601 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.612 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.614 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.624 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.626 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.637 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.638 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.649 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.651 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.651 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.652 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxI 2021-11-20T07:25:26.654 INFO:teuthology.task.ansible.out:vCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.663 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.675 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.687 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.689 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.690 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.701 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.703 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.704 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.705 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.716 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.718 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.729 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.734 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\ns 2021-11-20T07:25:26.736 INFO:teuthology.task.ansible.out:just@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.744 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.757 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.770 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.782 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.783 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.783 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.784 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.785 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.786 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph. 2021-11-20T07:25:26.790 INFO:teuthology.task.ansible.out:com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.797 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.809 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.819 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.821 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.831 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.833 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.834 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.835 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.836 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.848 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.850 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.861 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.863 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.873 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.875 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.886 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.887 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.899 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.901 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.902 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.912 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.914 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.925 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.937 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.939 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.948 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.950 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.960 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.962 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.973 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.984 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.986 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.996 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:26.999 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.008 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.010 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.021 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.022 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.033 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.035 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.046 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.047 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.048 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.058 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.060 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.071 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.073 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.084 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.086 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.095 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.097 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.108 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.110 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.119 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.121 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.134 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.145 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.146 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.157 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.161 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.169 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.172 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.181 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.183 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.194 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.195 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.206 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.208 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.219 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.221 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.230 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.250 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.252 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.252 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.273 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:27 +0000 (0:00:01.722) 0:01:46.341 ***** 2021-11-20T07:25:27.280 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.282 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.282 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.295 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.297 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.307 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.308 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.319 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.341 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.342 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.353 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.354 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.377 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:27 +0000 (0:00:00.104) 0:01:46.445 ***** 2021-11-20T07:25:27.384 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.386 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.396 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.398 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.439 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:27 +0000 (0:00:00.061) 0:01:46.507 ***** 2021-11-20T07:25:27.508 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-11-20T07:25:27.510 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:27.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:27 +0000 (0:00:00.090) 0:01:46.598 ***** 2021-11-20T07:25:27.663 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T07:25:27.665 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T07:25:27.811 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T07:25:27.853 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T07:25:28.001 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T07:25:28.043 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T07:25:28.190 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T07:25:28.244 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T07:25:28.393 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T07:25:28.436 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T07:25:28.583 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T07:25:28.625 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T07:25:28.783 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T07:25:28.827 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T07:25:28.974 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T07:25:29.017 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T07:25:29.173 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T07:25:29.216 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T07:25:29.367 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T07:25:29.410 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T07:25:29.567 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T07:25:29.599 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T07:25:29.757 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T07:25:29.800 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T07:25:29.950 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T07:25:29.995 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T07:25:30.143 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T07:25:30.185 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T07:25:30.343 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T07:25:30.386 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T07:25:30.544 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T07:25:30.577 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T07:25:30.745 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T07:25:30.777 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T07:25:30.944 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T07:25:30.966 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T07:25:31.133 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T07:25:31.166 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T07:25:31.323 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T07:25:31.356 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T07:25:31.523 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T07:25:31.545 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T07:25:31.714 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T07:25:31.736 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T07:25:31.913 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T07:25:31.925 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T07:25:32.102 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T07:25:32.124 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T07:25:32.290 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T07:25:32.322 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T07:25:32.490 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T07:25:32.512 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T07:25:32.681 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T07:25:32.702 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T07:25:32.870 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T07:25:32.892 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T07:25:33.060 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T07:25:33.093 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T07:25:33.249 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T07:25:33.281 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T07:25:33.451 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T07:25:33.484 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T07:25:33.644 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T07:25:33.666 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T07:25:33.835 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T07:25:33.868 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T07:25:34.034 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T07:25:34.066 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T07:25:34.234 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T07:25:34.265 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T07:25:34.423 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T07:25:34.509 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T07:25:34.641 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T07:25:34.664 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T07:25:34.833 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T07:25:34.871 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T07:25:35.029 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T07:25:35.072 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T07:25:35.230 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T07:25:36.604 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T07:25:36.837 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T07:25:38.192 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T07:25:38.474 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T07:25:39.834 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T07:25:40.105 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T07:25:41.404 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T07:25:41.688 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T07:25:42.920 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T07:25:43.321 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T07:25:44.481 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T07:25:44.680 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T07:25:44.848 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T07:25:44.892 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T07:25:45.061 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T07:25:45.083 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T07:25:45.263 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T07:25:45.275 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T07:25:45.455 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T07:25:45.467 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T07:25:45.646 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T07:25:45.668 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T07:25:45.838 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T07:25:45.860 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T07:25:46.042 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T07:25:46.052 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T07:25:46.233 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T07:25:46.245 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T07:25:46.424 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T07:25:46.447 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T07:25:46.504 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:46 +0000 (0:00:18.974) 0:02:05.572 ***** 2021-11-20T07:25:46.620 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T07:25:46.821 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T07:25:46.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:46 +0000 (0:00:00.372) 0:02:05.945 ***** 2021-11-20T07:25:47.076 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T07:25:47.079 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393146.606132-17905-84118572582490/source", "state": "file", "uid": 0} 2021-11-20T07:25:47.100 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.223) 0:02:06.168 ***** 2021-11-20T07:25:47.177 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi112.front.sepia.ceph.com] => { "msg": "Host smithi112.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T07:25:47.204 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.103) 0:02:06.272 ***** 2021-11-20T07:25:47.285 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T07:25:47.307 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.103) 0:02:06.375 ***** 2021-11-20T07:25:47.386 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T07:25:47.388 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:47.409 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.102) 0:02:06.477 ***** 2021-11-20T07:25:47.427 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T07:25:47.428 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393146.9713402-17964-127418259016113/source", "state": "file", "uid": 0} 2021-11-20T07:25:47.452 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.042) 0:02:06.520 ***** 2021-11-20T07:25:47.533 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T07:25:47.535 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => { "msg": "Host smithi071.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T07:25:47.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.104) 0:02:06.624 ***** 2021-11-20T07:25:47.638 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T07:25:47.639 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T07:25:47.662 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.105) 0:02:06.730 ***** 2021-11-20T07:25:47.732 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T07:25:47.734 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.087828", "end": "2021-11-20 07:25:47.708383", "rc": 0, "start": "2021-11-20 07:25:47.620555", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T07:25:47.754 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.092) 0:02:06.822 ***** 2021-11-20T07:25:47.773 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.018) 0:02:06.841 ***** 2021-11-20T07:25:47.856 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-20T07:25:47.858 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:47.881 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.108) 0:02:06.950 ***** 2021-11-20T07:25:47.952 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T07:25:47.954 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:47.978 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:47 +0000 (0:00:00.096) 0:02:07.046 ***** 2021-11-20T07:25:48.061 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.084 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.105) 0:02:07.152 ***** 2021-11-20T07:25:48.091 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T07:25:48.092 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.084725", "end": "2021-11-20 07:25:48.047489", "rc": 0, "start": "2021-11-20 07:25:47.962764", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T07:25:48.117 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.033) 0:02:07.185 ***** 2021-11-20T07:25:48.160 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.184 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.067) 0:02:07.253 ***** 2021-11-20T07:25:48.186 INFO:teuthology.task.ansible.out: 2021-11-20T07:25:48.205 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.230 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.042) 0:02:07.295 ***** 2021-11-20T07:25:48.256 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.284 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.051) 0:02:07.347 ***** 2021-11-20T07:25:48.297 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T07:25:48.298 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.320 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.041) 0:02:07.388 ***** 2021-11-20T07:25:48.359 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T07:25:48.361 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.394 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.073) 0:02:07.462 ***** 2021-11-20T07:25:48.396 INFO:teuthology.task.ansible.out: 2021-11-20T07:25:48.403 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-11-20T07:25:48.405 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.429 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.034) 0:02:07.497 ***** 2021-11-20T07:25:48.482 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi112.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:48.505 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.075) 0:02:07.573 ***** 2021-11-20T07:25:48.513 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T07:25:48.515 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.538 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.033) 0:02:07.607 ***** 2021-11-20T07:25:48.570 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T07:25:48.572 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.594 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.055) 0:02:07.662 ***** 2021-11-20T07:25:48.638 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.667 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.072) 0:02:07.735 ***** 2021-11-20T07:25:48.689 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T07:25:48.692 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.718 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.048) 0:02:07.784 ***** 2021-11-20T07:25:48.755 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T07:25:48.759 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.780 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.063) 0:02:07.848 ***** 2021-11-20T07:25:48.800 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:48.825 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.045) 0:02:07.893 ***** 2021-11-20T07:25:48.864 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-20T07:25:48.866 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:48.890 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.064) 0:02:07.957 ***** 2021-11-20T07:25:48.909 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T07:25:48.912 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:48.943 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:48 +0000 (0:00:00.053) 0:02:08.011 ***** 2021-11-20T07:25:48.985 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T07:25:48.987 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.011 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.067) 0:02:08.078 ***** 2021-11-20T07:25:49.029 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T07:25:49.030 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.055 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.044) 0:02:08.123 ***** 2021-11-20T07:25:49.084 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.107 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.052) 0:02:08.175 ***** 2021-11-20T07:25:49.137 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T07:25:49.139 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:49.163 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.056) 0:02:08.231 ***** 2021-11-20T07:25:49.192 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T07:25:49.194 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.224 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.061) 0:02:08.292 ***** 2021-11-20T07:25:49.245 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.271 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.046) 0:02:08.339 ***** 2021-11-20T07:25:49.305 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T07:25:49.307 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:49.329 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.058) 0:02:08.397 ***** 2021-11-20T07:25:49.382 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-20T07:25:49.384 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.407 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.078) 0:02:08.475 ***** 2021-11-20T07:25:49.415 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T07:25:49.417 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.444 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.036) 0:02:08.512 ***** 2021-11-20T07:25:49.483 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-20T07:25:49.485 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.507 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.063) 0:02:08.575 ***** 2021-11-20T07:25:49.526 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T07:25:49.528 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:49.550 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.042) 0:02:08.618 ***** 2021-11-20T07:25:49.579 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.603 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.053) 0:02:08.671 ***** 2021-11-20T07:25:49.622 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.645 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.042) 0:02:08.713 ***** 2021-11-20T07:25:49.678 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T07:25:49.680 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:49.707 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.061) 0:02:08.775 ***** 2021-11-20T07:25:49.727 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.750 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.043) 0:02:08.818 ***** 2021-11-20T07:25:49.783 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.808 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.057) 0:02:08.876 ***** 2021-11-20T07:25:49.825 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-20T07:25:49.827 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.851 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.043) 0:02:08.919 ***** 2021-11-20T07:25:49.894 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T07:25:49.897 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.930 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.076) 0:02:08.996 ***** 2021-11-20T07:25:49.953 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:49.994 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:49 +0000 (0:00:00.062) 0:02:09.059 ***** 2021-11-20T07:25:50.012 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T07:25:50.015 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.049 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.057) 0:02:09.117 ***** 2021-11-20T07:25:50.081 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:50.104 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.055) 0:02:09.172 ***** 2021-11-20T07:25:50.133 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T07:25:50.135 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.160 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.055) 0:02:09.228 ***** 2021-11-20T07:25:50.179 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T07:25:50.182 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.042) 0:02:09.271 ***** 2021-11-20T07:25:50.243 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.268 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.064) 0:02:09.336 ***** 2021-11-20T07:25:50.286 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T07:25:50.288 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.316 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.044) 0:02:09.380 ***** 2021-11-20T07:25:50.351 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.378 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.066) 0:02:09.446 ***** 2021-11-20T07:25:50.398 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T07:25:50.400 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.430 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.050) 0:02:09.496 ***** 2021-11-20T07:25:50.460 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:50.485 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.056) 0:02:09.553 ***** 2021-11-20T07:25:50.513 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T07:25:50.515 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.539 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.053) 0:02:09.606 ***** 2021-11-20T07:25:50.556 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T07:25:50.558 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.043) 0:02:09.650 ***** 2021-11-20T07:25:50.612 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T07:25:50.613 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.636 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.053) 0:02:09.704 ***** 2021-11-20T07:25:50.655 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T07:25:50.657 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.684 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.044) 0:02:09.749 ***** 2021-11-20T07:25:50.709 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T07:25:50.711 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.733 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.051) 0:02:09.801 ***** 2021-11-20T07:25:50.804 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T07:25:50.806 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T07:25:50.827 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.094) 0:02:09.895 ***** 2021-11-20T07:25:50.908 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T07:25:50.910 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:50.936 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.109) 0:02:10.004 ***** 2021-11-20T07:25:50.964 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-20T07:25:50.966 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:25:51.001 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:50 +0000 (0:00:00.064) 0:02:10.068 ***** 2021-11-20T07:25:51.011 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T07:25:51.013 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:51.036 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:51 +0000 (0:00:00.035) 0:02:10.104 ***** 2021-11-20T07:25:51.310 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:25:51.337 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:51 +0000 (0:00:00.300) 0:02:10.404 ***** 2021-11-20T07:25:51.794 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T07:25:51.867 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T07:25:52.232 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T07:25:52.264 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:52 +0000 (0:00:00.926) 0:02:11.331 ***** 2021-11-20T07:25:52.316 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T07:25:52.351 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:52 +0000 (0:00:00.087) 0:02:11.419 ***** 2021-11-20T07:25:52.805 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T07:25:52.807 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T07:25:52.870 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T07:25:53.253 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T07:25:53.277 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.926) 0:02:12.345 ***** 2021-11-20T07:25:53.329 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T07:25:53.354 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.077) 0:02:12.422 ***** 2021-11-20T07:25:53.362 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T07:25:53.364 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.034) 0:02:12.456 ***** 2021-11-20T07:25:53.438 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.465 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.074) 0:02:12.531 ***** 2021-11-20T07:25:53.475 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T07:25:53.477 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.510 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.046) 0:02:12.578 ***** 2021-11-20T07:25:53.555 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.579 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.069) 0:02:12.647 ***** 2021-11-20T07:25:53.597 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-20T07:25:53.599 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.621 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.041) 0:02:12.689 ***** 2021-11-20T07:25:53.649 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.673 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.052) 0:02:12.741 ***** 2021-11-20T07:25:53.692 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-20T07:25:53.694 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.717 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.043) 0:02:12.785 ***** 2021-11-20T07:25:53.759 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.782 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.064) 0:02:12.850 ***** 2021-11-20T07:25:53.800 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-20T07:25:53.802 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.814 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.816 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.838 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.056) 0:02:12.906 ***** 2021-11-20T07:25:53.867 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.869 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.878 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.880 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.906 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.066) 0:02:12.973 ***** 2021-11-20T07:25:53.913 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T07:25:53.915 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:53.942 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:53 +0000 (0:00:00.035) 0:02:13.009 ***** 2021-11-20T07:25:53.981 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.073) 0:02:13.082 ***** 2021-11-20T07:25:54.024 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T07:25:54.025 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.051 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.037) 0:02:13.120 ***** 2021-11-20T07:25:54.093 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.122 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.070) 0:02:13.190 ***** 2021-11-20T07:25:54.130 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-20T07:25:54.132 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.165 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.042) 0:02:13.232 ***** 2021-11-20T07:25:54.197 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.221 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.056) 0:02:13.289 ***** 2021-11-20T07:25:54.240 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-20T07:25:54.242 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.264 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.042) 0:02:13.331 ***** 2021-11-20T07:25:54.291 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.316 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.052) 0:02:13.384 ***** 2021-11-20T07:25:54.336 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.360 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.044) 0:02:13.428 ***** 2021-11-20T07:25:54.422 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:25:54.450 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:54 +0000 (0:00:00.089) 0:02:13.518 ***** 2021-11-20T07:25:55.521 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-20T07:25:55.523 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.874144", "end": "2021-11-20 07:25:55.500763", "rc": 0, "start": "2021-11-20 07:25:54.626619", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.966670", "end": "2021-11-20 07:25:55.500342", "rc": 0, "start": "2021-11-20 07:25:54.533672", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:25:55.545 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:55 +0000 (0:00:01.095) 0:02:14.613 ***** 2021-11-20T07:25:55.567 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:25:55 +0000 (0:00:00.021) 0:02:14.635 ***** 2021-11-20T07:26:00.011 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-20T07:26:00.013 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:00.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:04.467) 0:02:19.102 ***** 2021-11-20T07:26:00.104 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T07:26:00.106 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:00.129 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.095) 0:02:19.197 ***** 2021-11-20T07:26:00.150 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.020) 0:02:19.218 ***** 2021-11-20T07:26:00.190 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-20T07:26:00.191 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.215 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.064) 0:02:19.283 ***** 2021-11-20T07:26:00.221 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T07:26:00.223 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.246 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.031) 0:02:19.314 ***** 2021-11-20T07:26:00.286 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-20T07:26:00.288 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.310 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.063) 0:02:19.378 ***** 2021-11-20T07:26:00.318 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-20T07:26:00.321 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.037) 0:02:19.415 ***** 2021-11-20T07:26:00.375 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T07:26:00.378 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.400 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.052) 0:02:19.468 ***** 2021-11-20T07:26:00.418 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.447 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.047) 0:02:19.515 ***** 2021-11-20T07:26:00.517 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:00.541 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.093) 0:02:19.608 ***** 2021-11-20T07:26:00.907 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-11-20T07:26:00.909 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T07:26:00.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:00 +0000 (0:00:00.394) 0:02:20.003 ***** 2021-11-20T07:26:01.047 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T07:26:01.071 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:01 +0000 (0:00:00.135) 0:02:20.139 ***** 2021-11-20T07:26:02.824 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-20T07:26:02.826 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:02.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:02 +0000 (0:00:01.775) 0:02:21.915 ***** 2021-11-20T07:26:02.962 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:02.985 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:02 +0000 (0:00:00.138) 0:02:22.053 ***** 2021-11-20T07:26:03.409 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T07:26:03.412 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T07:26:03.538 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T07:26:03.877 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T07:26:04.003 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T07:26:04.320 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T07:26:04.346 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:04 +0000 (0:00:01.359) 0:02:23.413 ***** 2021-11-20T07:26:04.470 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T07:26:04.506 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:04 +0000 (0:00:00.161) 0:02:23.574 ***** 2021-11-20T07:26:04.534 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:26:04.559 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:04 +0000 (0:00:00.052) 0:02:23.627 ***** 2021-11-20T07:26:04.709 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:26:04.733 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:04 +0000 (0:00:00.173) 0:02:23.801 ***** 2021-11-20T07:26:05.064 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T07:26:05.065 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T07:26:05.259 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T07:26:05.492 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T07:26:05.704 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T07:26:06.011 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393165.5136368-20825-105070160154231/source", "state": "file", "uid": 0} 2021-11-20T07:26:06.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:01.340) 0:02:25.141 ***** 2021-11-20T07:26:06.112 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T07:26:06.113 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T07:26:06.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.059) 0:02:25.201 ***** 2021-11-20T07:26:06.204 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T07:26:06.206 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393165.7407124-20851-136857481494192/source", "state": "file", "uid": 0} 2021-11-20T07:26:06.241 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.106) 0:02:25.308 ***** 2021-11-20T07:26:06.280 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T07:26:06.282 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T07:26:06.306 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.065) 0:02:25.374 ***** 2021-11-20T07:26:06.323 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T07:26:06.324 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T07:26:06.347 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.040) 0:02:25.415 ***** 2021-11-20T07:26:06.396 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T07:26:06.398 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.408 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.418 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.445 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.098) 0:02:25.513 ***** 2021-11-20T07:26:06.505 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T07:26:06.506 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T07:26:06.531 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.083) 0:02:25.597 ***** 2021-11-20T07:26:06.580 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T07:26:06.582 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.592 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.602 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.613 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:06.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:06 +0000 (0:00:00.108) 0:02:25.706 ***** 2021-11-20T07:26:08.255 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:08.279 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:08 +0000 (0:00:01.641) 0:02:27.347 ***** 2021-11-20T07:26:08.313 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:08.342 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:08 +0000 (0:00:00.057) 0:02:27.405 ***** 2021-11-20T07:26:08.451 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T07:26:08.453 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:08.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:08 +0000 (0:00:00.140) 0:02:27.546 ***** 2021-11-20T07:26:08.507 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2021-11-20T07:26:08.508 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:08.529 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:08 +0000 (0:00:00.051) 0:02:27.597 ***** 2021-11-20T07:26:08.812 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T07:26:08.837 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:08 +0000 (0:00:00.307) 0:02:27.905 ***** 2021-11-20T07:26:09.002 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T07:26:09.028 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.191) 0:02:28.096 ***** 2021-11-20T07:26:09.322 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T07:26:09.324 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T07:26:09.346 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.317) 0:02:28.414 ***** 2021-11-20T07:26:09.395 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T07:26:09.397 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:09.408 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:09.431 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.084) 0:02:28.499 ***** 2021-11-20T07:26:09.525 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T07:26:09.527 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T07:26:09.554 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.123) 0:02:28.622 ***** 2021-11-20T07:26:09.595 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi112.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:09.606 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T07:26:09.629 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.075) 0:02:28.697 ***** 2021-11-20T07:26:09.646 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-20T07:26:09.648 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:26:09.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.041) 0:02:28.739 ***** 2021-11-20T07:26:09.826 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:26:09.850 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.178) 0:02:28.917 ***** 2021-11-20T07:26:09.908 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T07:26:09.910 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T07:26:09.933 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.083) 0:02:29.001 ***** 2021-11-20T07:26:09.961 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T07:26:09.963 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:09.984 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:09 +0000 (0:00:00.050) 0:02:29.052 ***** 2021-11-20T07:26:10.087 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T07:26:10.089 INFO:teuthology.task.ansible.out:fatal: [smithi112.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T07:26:10.118 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:10 +0000 (0:00:00.133) 0:02:29.186 ***** 2021-11-20T07:26:10.157 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T07:26:10.159 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:26:10.180 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:10 +0000 (0:00:00.061) 0:02:29.247 ***** 2021-11-20T07:26:10.471 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T07:26:10.495 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:10 +0000 (0:00:00.315) 0:02:29.563 ***** 2021-11-20T07:26:10.664 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T07:26:10.690 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:10 +0000 (0:00:00.194) 0:02:29.757 ***** 2021-11-20T07:26:11.174 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T07:26:11.175 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 07:22:54 UTC", "ActiveEnterTimestampMonotonic": "19284406", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount nss-lookup.target network-online.target sysinit.target system.slice local-fs.target basic.target time-sync.target -.mount systemd-tmpfiles-setup.service systemd-journald.socket var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 07:22:54 UTC", "AssertTimestampMonotonic": "19267712", "Before": "xdm.service getty@tty1.service shutdown.target plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 07:22:54 UTC", "Co 2021-11-20T07:26:11.177 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19267712", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1287", "ExecMainStartTimestamp": "Sat 2021-11-20 07:22:54 UTC", "ExecMainStartTimestampMonotonic": "19284363", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 07:22:54 UTC] ; stop_time=[n/a] ; pid=1287 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 07:22:54 UTC", "InactiveExitTimestampMonotonic": "192844 2021-11-20T07:26:11.178 INFO:teuthology.task.ansible.out:06", "InvocationID": "f1c8d619af7444069b4aa10ce12b34ef", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1287", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T07:26:11.180 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 07:22:54 UTC", "StateChangeTimestampMonotonic": "19284406", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 07:22:54 UTC", "WatchdogTimestampMonotonic": "19284405", "WatchdogUSec": "0"}} 2021-11-20T07:26:11.198 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:11 +0000 (0:00:00.508) 0:02:30.266 ***** 2021-11-20T07:26:11.237 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 07:22:54 UTC", "ActiveEnterTimestampMonotonic": "19549186", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target var-run.mount tmp.mount time-sync.target systemd-journald.socket basic.target local-fs.target sysinit.target systemd-tmpfiles-setup.service system.slice -.mount network-online.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 07:22:54 UTC", "AssertTimestampMonotonic": "19525762", "Before": "shutdown.target getty@tty1.service plymouth-quit.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 07:22:54 UTC", "Co 2021-11-20T07:26:11.241 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19525761", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1278", "ExecMainStartTimestamp": "Sat 2021-11-20 07:22:54 UTC", "ExecMainStartTimestampMonotonic": "19548762", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 07:22:54 UTC] ; stop_time=[n/a] ; pid=1278 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 07:22:54 UTC", "InactiveExitTimestampMonotonic": "195491 2021-11-20T07:26:11.243 INFO:teuthology.task.ansible.out:86", "InvocationID": "a53bcd5ebd21461698cf0b6851fbd9dd", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1278", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T07:26:11.245 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 07:22:54 UTC", "StateChangeTimestampMonotonic": "19549186", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 07:22:54 UTC", "WatchdogTimestampMonotonic": "19549167", "WatchdogUSec": "0"}} 2021-11-20T07:26:11.249 INFO:teuthology.task.ansible.out: 2021-11-20T07:26:11.281 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:11 +0000 (0:00:00.082) 0:02:30.349 ***** 2021-11-20T07:26:11.393 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-20T07:26:11.394 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001693", "end": "2021-11-20 07:26:11.368198", "rc": 0, "start": "2021-11-20 07:26:11.366505", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T07:26:11.427 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:11 +0000 (0:00:00.146) 0:02:30.495 ***** 2021-11-20T07:26:11.531 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001809", "end": "2021-11-20 07:26:11.510353", "rc": 0, "start": "2021-11-20 07:26:11.508544", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T07:26:11.567 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:11 +0000 (0:00:00.139) 0:02:30.635 ***** 2021-11-20T07:26:13.304 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-20T07:26:13.306 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:13.327 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:13 +0000 (0:00:01.760) 0:02:32.395 ***** 2021-11-20T07:26:13.367 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:26:13.392 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:13 +0000 (0:00:00.065) 0:02:32.460 ***** 2021-11-20T07:26:52.194 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-20T07:26:52.196 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T07:26:52.220 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:52 +0000 (0:00:38.828) 0:03:11.288 ***** 2021-11-20T07:26:52.229 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T07:26:52.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:52 +0000 (0:00:00.035) 0:03:11.323 ***** 2021-11-20T07:26:52.910 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T07:26:52.930 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:52 +0000 (0:00:00.674) 0:03:11.998 ***** 2021-11-20T07:26:52.938 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T07:26:52.965 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:26:52 +0000 (0:00:00.034) 0:03:12.033 ***** 2021-11-20T07:26:52.968 INFO:teuthology.task.ansible.out: 2021-11-20T07:27:08.663 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.504206", "end": "2021-11-20 07:27:08.639810", "failed_when_result": false, "rc": 0, "start": "2021-11-20 07:26:53.135604", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T07:27:08.683 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:08 +0000 (0:00:15.717) 0:03:27.751 ***** 2021-11-20T07:27:08.859 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.730763", "end": "2021-11-20 07:27:08.829956", "failed_when_result": false, "rc": 0, "start": "2021-11-20 07:26:53.099193", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T07:27:08.886 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:08 +0000 (0:00:00.198) 0:03:27.949 ***** 2021-11-20T07:27:09.178 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T07:27:09.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.321) 0:03:28.271 ***** 2021-11-20T07:27:09.365 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T07:27:09.396 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.193) 0:03:28.464 ***** 2021-11-20T07:27:09.414 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T07:27:09.416 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010769", "end": "2021-11-20 07:27:09.386721", "rc": 0, "start": "2021-11-20 07:27:09.375952", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:09.443 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.046) 0:03:28.511 ***** 2021-11-20T07:27:09.445 INFO:teuthology.task.ansible.out: 2021-11-20T07:27:09.598 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010805", "end": "2021-11-20 07:27:09.573955", "rc": 0, "start": "2021-11-20 07:27:09.563150", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:09.622 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.178) 0:03:28.690 ***** 2021-11-20T07:27:09.640 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.008371", "end": "2021-11-20 07:27:09.621433", "rc": 0, "start": "2021-11-20 07:27:09.613062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:09.667 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.045) 0:03:28.735 ***** 2021-11-20T07:27:09.826 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012835", "end": "2021-11-20 07:27:09.803967", "rc": 0, "start": "2021-11-20 07:27:09.791132", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:09.850 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:09 +0000 (0:00:00.183) 0:03:28.918 ***** 2021-11-20T07:27:25.171 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.303796", "end": "2021-11-20 07:27:25.144969", "rc": 0, "start": "2021-11-20 07:27:09.841173", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:25.196 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:15.345) 0:03:44.264 ***** 2021-11-20T07:27:25.395 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T07:27:25.418 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.222) 0:03:44.486 ***** 2021-11-20T07:27:25.460 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T07:27:25.486 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.064) 0:03:44.551 ***** 2021-11-20T07:27:25.506 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.459466", "end": "2021-11-20 07:27:25.479492", "rc": 0, "start": "2021-11-20 07:27:10.020026", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:25.529 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.046) 0:03:44.597 ***** 2021-11-20T07:27:25.536 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T07:27:25.542 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T07:27:25.563 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.034) 0:03:44.631 ***** 2021-11-20T07:27:25.612 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T07:27:25.614 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T07:27:25.615 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T07:27:25.637 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.073) 0:03:44.705 ***** 2021-11-20T07:27:25.686 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T07:27:25.689 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T07:27:25.691 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T07:27:25.713 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.076) 0:03:44.781 ***** 2021-11-20T07:27:25.732 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T07:27:25.764 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.051) 0:03:44.832 ***** 2021-11-20T07:27:25.806 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T07:27:25.829 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.064) 0:03:44.897 ***** 2021-11-20T07:27:25.869 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T07:27:25.870 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T07:27:25.892 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.062) 0:03:44.960 ***** 2021-11-20T07:27:25.912 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T07:27:25.914 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T07:27:25.938 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.046) 0:03:45.006 ***** 2021-11-20T07:27:25.946 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T07:27:25.948 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T07:27:25.950 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T07:27:25.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:25 +0000 (0:00:00.034) 0:03:45.041 ***** 2021-11-20T07:27:26.030 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T07:27:26.032 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T07:27:26.034 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T07:27:26.066 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.091) 0:03:45.133 ***** 2021-11-20T07:27:26.125 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T07:27:26.127 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T07:27:26.150 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.084) 0:03:45.218 ***** 2021-11-20T07:27:26.178 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:26.200 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.050) 0:03:45.268 ***** 2021-11-20T07:27:26.228 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T07:27:26.229 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:26.251 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.050) 0:03:45.319 ***** 2021-11-20T07:27:26.258 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T07:27:26.260 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T07:27:26.283 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.032) 0:03:45.351 ***** 2021-11-20T07:27:26.476 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi071.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T07:27:26.498 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.214) 0:03:45.566 ***** 2021-11-20T07:27:26.520 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.021) 0:03:45.588 ***** 2021-11-20T07:27:26.561 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:26.597 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.074) 0:03:45.662 ***** 2021-11-20T07:27:26.623 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:26.646 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.051) 0:03:45.714 ***** 2021-11-20T07:27:26.876 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T07:27:26.878 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T07:27:26.900 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:26 +0000 (0:00:00.254) 0:03:45.968 ***** 2021-11-20T07:27:27.066 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T07:27:27.068 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T07:27:27.092 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:27 +0000 (0:00:00.191) 0:03:46.160 ***** 2021-11-20T07:27:27.132 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi071"}, "changed": false} 2021-11-20T07:27:27.154 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:27 +0000 (0:00:00.061) 0:03:46.221 ***** 2021-11-20T07:27:27.455 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T07:27:27.457 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T07:27:27.489 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:27 +0000 (0:00:00.333) 0:03:46.555 ***** 2021-11-20T07:27:27.548 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-11-20T07:27:27.552 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi112"}, "changed": false} 2021-11-20T07:27:27.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:27 +0000 (0:00:00.084) 0:03:46.640 ***** 2021-11-20T07:27:27.976 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-20T07:27:27.979 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi071.front.sepia.ceph.com", "ansible_hostname": "smithi071", "ansible_nodename": "smithi071"}, "changed": true, "name": "smithi071"} 2021-11-20T07:27:28.003 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:27 +0000 (0:00:00.430) 0:03:47.071 ***** 2021-11-20T07:27:28.022 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi112.front.sepia.ceph.com", "ansible_hostname": "smithi112", "ansible_nodename": "smithi112"}, "changed": true, "name": "smithi112"} 2021-11-20T07:27:28.046 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:28 +0000 (0:00:00.043) 0:03:47.114 ***** 2021-11-20T07:27:28.473 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-20T07:27:28.475 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T07:27:28.502 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:28 +0000 (0:00:00.455) 0:03:47.570 ***** 2021-11-20T07:27:28.547 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T07:27:28.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:28 +0000 (0:00:00.076) 0:03:47.646 ***** 2021-11-20T07:27:28.968 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T07:27:28.992 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:28 +0000 (0:00:00.413) 0:03:48.059 ***** 2021-11-20T07:27:29.054 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T07:27:29.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:29 +0000 (0:00:00.093) 0:03:48.153 ***** 2021-11-20T07:27:29.516 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T07:27:29.539 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:29 +0000 (0:00:00.453) 0:03:48.606 ***** 2021-11-20T07:27:29.622 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T07:27:29.652 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:29 +0000 (0:00:00.112) 0:03:48.719 ***** 2021-11-20T07:27:30.037 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T07:27:30.039 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T07:27:30.063 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.412) 0:03:49.131 ***** 2021-11-20T07:27:30.145 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T07:27:30.169 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.106) 0:03:49.237 ***** 2021-11-20T07:27:30.268 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T07:27:30.270 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T07:27:30.301 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.131) 0:03:49.369 ***** 2021-11-20T07:27:30.374 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T07:27:30.401 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.098) 0:03:49.467 ***** 2021-11-20T07:27:30.495 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T07:27:30.497 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T07:27:30.528 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.128) 0:03:49.596 ***** 2021-11-20T07:27:30.607 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T07:27:30.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.109) 0:03:49.706 ***** 2021-11-20T07:27:30.759 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T07:27:30.760 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T07:27:30.785 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.146) 0:03:49.853 ***** 2021-11-20T07:27:30.880 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T07:27:30.904 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:30 +0000 (0:00:00.118) 0:03:49.972 ***** 2021-11-20T07:27:31.301 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T07:27:31.325 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:31 +0000 (0:00:00.420) 0:03:50.393 ***** 2021-11-20T07:27:31.390 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T07:27:31.418 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:31 +0000 (0:00:00.092) 0:03:50.486 ***** 2021-11-20T07:27:31.817 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T07:27:31.841 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:31 +0000 (0:00:00.423) 0:03:50.909 ***** 2021-11-20T07:27:31.914 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T07:27:31.937 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:31 +0000 (0:00:00.095) 0:03:51.005 ***** 2021-11-20T07:27:34.469 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T07:27:34.472 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.432786", "end": "2021-11-20 07:27:34.445726", "rc": 0, "start": "2021-11-20 07:27:32.012940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:34.500 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:02.562) 0:03:53.567 ***** 2021-11-20T07:27:34.667 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.525901", "end": "2021-11-20 07:27:34.635199", "rc": 0, "start": "2021-11-20 07:27:32.109298", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:34.693 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.193) 0:03:53.761 ***** 2021-11-20T07:27:34.722 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002637", "end": "2021-11-20 07:27:34.692144", "rc": 0, "start": "2021-11-20 07:27:34.689507", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:34.747 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.054) 0:03:53.815 ***** 2021-11-20T07:27:34.789 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T07:27:34.810 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.062) 0:03:53.878 ***** 2021-11-20T07:27:34.870 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T07:27:34.872 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T07:27:34.893 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.083) 0:03:53.961 ***** 2021-11-20T07:27:34.911 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T07:27:34.912 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002637", "end": "2021-11-20 07:27:34.882557", "rc": 0, "start": "2021-11-20 07:27:34.879920", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:27:34.937 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.042) 0:03:54.004 ***** 2021-11-20T07:27:34.974 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:27:34.976 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T07:27:35.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:34 +0000 (0:00:00.066) 0:03:54.070 ***** 2021-11-20T07:27:35.074 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T07:27:35.097 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:35 +0000 (0:00:00.095) 0:03:54.165 ***** 2021-11-20T07:27:35.372 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T07:27:35.376 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T07:27:35.578 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T07:27:35.830 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T07:27:36.042 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T07:27:36.275 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T07:27:36.483 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T07:27:36.791 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393256.3011267-4920-226805101781345/source", "state": "file", "uid": 0} 2021-11-20T07:27:36.816 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:36 +0000 (0:00:01.718) 0:03:55.884 ***** 2021-11-20T07:27:37.020 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393256.5186503-4958-89401207842406/source", "state": "file", "uid": 0} 2021-11-20T07:27:37.049 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:37 +0000 (0:00:00.232) 0:03:56.117 ***** 2021-11-20T07:27:37.304 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-20T07:27:37.306 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T07:27:37.557 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T07:27:37.848 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393257.3547878-5380-52521344008482/source", "state": "file", "uid": 0} 2021-11-20T07:27:37.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:37 +0000 (0:00:00.825) 0:03:56.942 ***** 2021-11-20T07:27:38.055 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393257.586564-5448-94458331869059/source", "state": "file", "uid": 0} 2021-11-20T07:27:38.081 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:38 +0000 (0:00:00.206) 0:03:57.149 ***** 2021-11-20T07:27:38.357 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T07:27:38.359 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T07:27:38.559 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T07:27:38.797 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T07:27:39.010 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T07:27:39.254 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T07:27:39.458 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T07:27:39.750 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393259.2834897-5610-150742861911980/source", "state": "file", "uid": 0} 2021-11-20T07:27:39.775 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:39 +0000 (0:00:01.694) 0:03:58.844 ***** 2021-11-20T07:27:39.953 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393259.48436-5677-241779051950508/source", "state": "file", "uid": 0} 2021-11-20T07:27:39.986 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:39 +0000 (0:00:00.210) 0:03:59.054 ***** 2021-11-20T07:27:40.245 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-20T07:27:40.247 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.255341", "end": "2021-11-20 07:27:40.214425", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 07:27:39.959084", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2021-11-20T07:27:40.248 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2021-11-20T07:27:40.270 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:40 +0000 (0:00:00.284) 0:03:59.338 ***** 2021-11-20T07:27:40.457 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.262423", "end": "2021-11-20 07:27:40.434279", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 07:27:40.171856", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T07:27:40.483 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:40 +0000 (0:00:00.212) 0:03:59.551 ***** 2021-11-20T07:27:47.616 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-20T07:27:47.618 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:27:47.646 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:47 +0000 (0:00:07.161) 0:04:06.713 ***** 2021-11-20T07:27:47.653 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:27:47.678 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:47 +0000 (0:00:00.033) 0:04:06.746 ***** 2021-11-20T07:27:48.466 INFO:teuthology.task.ansible.out: 2021-11-20T07:27:48.467 INFO:teuthology.task.ansible.out:TASK [testnode : Install GPG keys] ********************************************* changed: [smithi112.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T07:27:48.479 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T07:27:48.934 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T07:27:48.955 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T07:27:50.295 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T07:27:50.319 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:02.640) 0:04:09.387 ***** 2021-11-20T07:27:50.464 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T07:27:50.495 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.175) 0:04:09.563 ***** 2021-11-20T07:27:50.722 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T07:27:50.724 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.180228", "end": "2021-11-20 07:27:50.704849", "rc": 0, "start": "2021-11-20 07:27:50.524621", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T07:27:50.749 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.254) 0:04:09.817 ***** 2021-11-20T07:27:50.794 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:27:50.795 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T07:27:50.819 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.069) 0:04:09.886 ***** 2021-11-20T07:27:50.861 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T07:27:50.863 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:50.887 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.068) 0:04:09.955 ***** 2021-11-20T07:27:50.906 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T07:27:50.909 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.185528", "end": "2021-11-20 07:27:50.884475", "rc": 0, "start": "2021-11-20 07:27:50.698947", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T07:27:50.932 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.045) 0:04:10.000 ***** 2021-11-20T07:27:50.941 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:50.968 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.035) 0:04:10.036 ***** 2021-11-20T07:27:50.976 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:27:50.978 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T07:27:51.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:50 +0000 (0:00:00.033) 0:04:10.070 ***** 2021-11-20T07:27:51.040 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T07:27:51.042 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:51.063 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.060) 0:04:10.131 ***** 2021-11-20T07:27:51.102 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:51.128 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.064) 0:04:10.195 ***** 2021-11-20T07:27:51.169 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T07:27:51.171 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007272", "end": "2021-11-20 07:27:51.149924", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 07:27:51.142652", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T07:27:51.195 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.068) 0:04:10.263 ***** 2021-11-20T07:27:51.224 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T07:27:51.226 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:51.246 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.050) 0:04:10.314 ***** 2021-11-20T07:27:51.326 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T07:27:51.328 INFO:teuthology.task.ansible.out:fatal: [smithi112.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007496", "end": "2021-11-20 07:27:51.308116", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 07:27:51.300620", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T07:27:51.352 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.105) 0:04:10.420 ***** 2021-11-20T07:27:51.390 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T07:27:51.392 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:27:51.417 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:27:51 +0000 (0:00:00.064) 0:04:10.485 ***** 2021-11-20T07:28:09.054 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T07:28:09.076 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:09 +0000 (0:00:17.659) 0:04:28.144 ***** 2021-11-20T07:28:09.176 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T07:28:09.201 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:09 +0000 (0:00:00.125) 0:04:28.269 ***** 2021-11-20T07:28:12.251 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T07:28:12.282 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:12 +0000 (0:00:03.080) 0:04:31.350 ***** 2021-11-20T07:28:12.305 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T07:28:12.328 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:12 +0000 (0:00:00.045) 0:04:31.396 ***** 2021-11-20T07:28:21.737 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:21.739 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T07:28:21.761 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:21 +0000 (0:00:09.433) 0:04:40.829 ***** 2021-11-20T07:28:21.801 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:21.823 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:21 +0000 (0:00:00.062) 0:04:40.891 ***** 2021-11-20T07:28:21.984 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:21.989 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T07:28:22.011 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:22 +0000 (0:00:00.186) 0:04:41.077 ***** 2021-11-20T07:28:22.039 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-20T07:28:22.042 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:22.066 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:22 +0000 (0:00:00.053) 0:04:41.131 ***** 2021-11-20T07:28:23.950 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:28:23.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:23 +0000 (0:00:01.910) 0:04:43.042 ***** 2021-11-20T07:28:24.249 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:28:24.273 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:24 +0000 (0:00:00.299) 0:04:43.341 ***** 2021-11-20T07:28:32.626 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T07:28:32.627 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T07:28:32.649 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:32 +0000 (0:00:08.376) 0:04:51.717 ***** 2021-11-20T07:28:32.827 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T07:28:32.852 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:32 +0000 (0:00:00.203) 0:04:51.920 ***** 2021-11-20T07:28:32.976 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T07:28:32.977 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service sysinit.target system.slice dbus.service basic.target dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service nftables.service ipset.service iptables.service shutdown.target ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T07:28:32.979 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T07:28:32.980 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T07:28:32.982 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T07:28:33.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:32 +0000 (0:00:00.149) 0:04:52.070 ***** 2021-11-20T07:28:33.170 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket polkit.service dbus.service system.slice sysinit.target basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ebtables.service ipset.service ip6tables.service iptables.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T07:28:33.172 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T07:28:33.174 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T07:28:33.175 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T07:28:33.196 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.193) 0:04:52.264 ***** 2021-11-20T07:28:33.291 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-20T07:28:33.293 INFO:teuthology.task.ansible.out:fatal: [smithi112.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T07:28:33.315 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.119) 0:04:52.383 ***** 2021-11-20T07:28:33.491 INFO:teuthology.task.ansible.out:fatal: [smithi071.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T07:28:33.515 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.199) 0:04:52.583 ***** 2021-11-20T07:28:33.912 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-20T07:28:33.914 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T07:28:33.934 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.419) 0:04:53.002 ***** 2021-11-20T07:28:33.941 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T07:28:33.966 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.031) 0:04:53.034 ***** 2021-11-20T07:28:33.973 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-20T07:28:33.975 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:33.999 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:33 +0000 (0:00:00.032) 0:04:53.067 ***** 2021-11-20T07:28:34.007 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.033) 0:04:53.101 ***** 2021-11-20T07:28:34.040 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T07:28:34.041 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.031) 0:04:53.133 ***** 2021-11-20T07:28:34.073 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.097 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.165 ***** 2021-11-20T07:28:34.103 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-20T07:28:34.104 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.035) 0:04:53.201 ***** 2021-11-20T07:28:34.141 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.165 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.233 ***** 2021-11-20T07:28:34.173 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T07:28:34.175 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.201 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.035) 0:04:53.269 ***** 2021-11-20T07:28:34.209 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.231 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.300 ***** 2021-11-20T07:28:34.238 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-20T07:28:34.240 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.264 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.332 ***** 2021-11-20T07:28:34.271 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.295 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.362 ***** 2021-11-20T07:28:34.301 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.324 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.392 ***** 2021-11-20T07:28:34.331 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.355 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.423 ***** 2021-11-20T07:28:34.362 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-20T07:28:34.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.454 ***** 2021-11-20T07:28:34.416 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.484 ***** 2021-11-20T07:28:34.423 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-20T07:28:34.425 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.517 ***** 2021-11-20T07:28:34.456 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.481 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.031) 0:04:53.548 ***** 2021-11-20T07:28:34.487 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T07:28:34.489 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.512 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.031) 0:04:53.580 ***** 2021-11-20T07:28:34.520 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.544 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.613 ***** 2021-11-20T07:28:34.552 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-11-20T07:28:34.553 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.033) 0:04:53.646 ***** 2021-11-20T07:28:34.587 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.598 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.599 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.624 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.045) 0:04:53.692 ***** 2021-11-20T07:28:34.646 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.022) 0:04:53.714 ***** 2021-11-20T07:28:34.654 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T07:28:34.679 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.746 ***** 2021-11-20T07:28:34.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T07:28:34.688 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.712 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.779 ***** 2021-11-20T07:28:34.721 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.747 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.032) 0:04:53.812 ***** 2021-11-20T07:28:34.751 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T07:28:34.779 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.034) 0:04:53.847 ***** 2021-11-20T07:28:34.789 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-20T07:28:34.791 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.815 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.035) 0:04:53.883 ***** 2021-11-20T07:28:34.822 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-20T07:28:34.824 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.031) 0:04:53.914 ***** 2021-11-20T07:28:34.854 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.945 ***** 2021-11-20T07:28:34.884 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-20T07:28:34.887 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.908 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.030) 0:04:53.976 ***** 2021-11-20T07:28:34.916 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.942 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.033) 0:04:54.009 ***** 2021-11-20T07:28:34.950 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T07:28:34.952 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.978 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:34 +0000 (0:00:00.036) 0:04:54.046 ***** 2021-11-20T07:28:34.997 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:34.999 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:35.020 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.041) 0:04:54.088 ***** 2021-11-20T07:28:35.027 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T07:28:35.029 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.052 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.120 ***** 2021-11-20T07:28:35.066 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:35.069 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.091 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.038) 0:04:54.159 ***** 2021-11-20T07:28:35.098 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T07:28:35.100 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.124 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.192 ***** 2021-11-20T07:28:35.131 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T07:28:35.133 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.156 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.224 ***** 2021-11-20T07:28:35.162 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:35.164 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.187 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.031) 0:04:54.255 ***** 2021-11-20T07:28:35.195 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T07:28:35.196 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.220 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.288 ***** 2021-11-20T07:28:35.228 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T07:28:35.229 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.042) 0:04:54.330 ***** 2021-11-20T07:28:35.272 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T07:28:35.274 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.297 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.034) 0:04:54.365 ***** 2021-11-20T07:28:35.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T07:28:35.307 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.331 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.033) 0:04:54.399 ***** 2021-11-20T07:28:35.339 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T07:28:35.342 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.365 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.034) 0:04:54.433 ***** 2021-11-20T07:28:35.373 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T07:28:35.375 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.409 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.043) 0:04:54.477 ***** 2021-11-20T07:28:35.420 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.040) 0:04:54.517 ***** 2021-11-20T07:28:35.458 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T07:28:35.459 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.550 ***** 2021-11-20T07:28:35.490 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T07:28:35.492 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.516 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.034) 0:04:54.584 ***** 2021-11-20T07:28:35.523 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T07:28:35.525 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.548 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.616 ***** 2021-11-20T07:28:35.556 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-20T07:28:35.558 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.033) 0:04:54.650 ***** 2021-11-20T07:28:35.590 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T07:28:35.591 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.615 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.682 ***** 2021-11-20T07:28:35.624 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-20T07:28:35.625 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.649 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.034) 0:04:54.717 ***** 2021-11-20T07:28:35.658 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-20T07:28:35.659 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.682 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.033) 0:04:54.750 ***** 2021-11-20T07:28:35.689 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.713 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.031) 0:04:54.781 ***** 2021-11-20T07:28:35.721 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.745 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.031) 0:04:54.813 ***** 2021-11-20T07:28:35.751 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.777 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.032) 0:04:54.845 ***** 2021-11-20T07:28:35.784 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T07:28:35.786 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.813 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.035) 0:04:54.881 ***** 2021-11-20T07:28:35.819 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T07:28:35.821 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.843 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.029) 0:04:54.911 ***** 2021-11-20T07:28:35.849 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T07:28:35.850 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.875 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.030) 0:04:54.941 ***** 2021-11-20T07:28:35.880 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T07:28:35.881 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.030) 0:04:54.972 ***** 2021-11-20T07:28:35.912 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.031) 0:04:55.004 ***** 2021-11-20T07:28:35.945 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:35.969 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.033) 0:04:55.037 ***** 2021-11-20T07:28:35.978 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T07:28:35.979 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.004 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:35 +0000 (0:00:00.034) 0:04:55.072 ***** 2021-11-20T07:28:36.011 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T07:28:36.013 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.036 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.032) 0:04:55.104 ***** 2021-11-20T07:28:36.049 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T07:28:36.051 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.037) 0:04:55.141 ***** 2021-11-20T07:28:36.081 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T07:28:36.083 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.106 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.032) 0:04:55.174 ***** 2021-11-20T07:28:36.118 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T07:28:36.119 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.152 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.046) 0:04:55.220 ***** 2021-11-20T07:28:36.163 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T07:28:36.165 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.037) 0:04:55.258 ***** 2021-11-20T07:28:36.199 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T07:28:36.201 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.226 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.036) 0:04:55.294 ***** 2021-11-20T07:28:36.233 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T07:28:36.235 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.258 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.031) 0:04:55.326 ***** 2021-11-20T07:28:36.266 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T07:28:36.268 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.294 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.033) 0:04:55.360 ***** 2021-11-20T07:28:36.300 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:36.302 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.325 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.033) 0:04:55.393 ***** 2021-11-20T07:28:36.342 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T07:28:36.344 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.365 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.040) 0:04:55.433 ***** 2021-11-20T07:28:36.372 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T07:28:36.373 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.396 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.031) 0:04:55.464 ***** 2021-11-20T07:28:36.404 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:36.405 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.428 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.032) 0:04:55.496 ***** 2021-11-20T07:28:36.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:36.436 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.030) 0:04:55.527 ***** 2021-11-20T07:28:36.467 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T07:28:36.468 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.491 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.032) 0:04:55.559 ***** 2021-11-20T07:28:36.498 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T07:28:36.500 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.030) 0:04:55.590 ***** 2021-11-20T07:28:36.529 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T07:28:36.531 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.035) 0:04:55.625 ***** 2021-11-20T07:28:36.566 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T07:28:36.567 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.591 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.033) 0:04:55.659 ***** 2021-11-20T07:28:36.599 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T07:28:36.601 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.631 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.038) 0:04:55.698 ***** 2021-11-20T07:28:36.660 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T07:28:36.662 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:36.685 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.055) 0:04:55.753 ***** 2021-11-20T07:28:36.818 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T07:28:36.842 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.156) 0:04:55.910 ***** 2021-11-20T07:28:36.902 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T07:28:36.925 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:36 +0000 (0:00:00.083) 0:04:55.993 ***** 2021-11-20T07:28:37.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T07:28:37.063 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:28:37.088 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.162) 0:04:56.156 ***** 2021-11-20T07:28:37.149 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:28:37.175 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.087) 0:04:56.243 ***** 2021-11-20T07:28:37.288 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T07:28:37.290 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T07:28:37.314 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.138) 0:04:56.381 ***** 2021-11-20T07:28:37.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T07:28:37.343 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.363 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.050) 0:04:56.431 ***** 2021-11-20T07:28:37.382 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T07:28:37.384 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T07:28:37.406 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.042) 0:04:56.474 ***** 2021-11-20T07:28:37.413 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T07:28:37.418 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.439 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.507 ***** 2021-11-20T07:28:37.447 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T07:28:37.449 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.471 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.539 ***** 2021-11-20T07:28:37.479 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T07:28:37.481 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.505 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.572 ***** 2021-11-20T07:28:37.512 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T07:28:37.514 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.536 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.031) 0:04:56.604 ***** 2021-11-20T07:28:37.543 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.569 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.031) 0:04:56.636 ***** 2021-11-20T07:28:37.576 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T07:28:37.578 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.602 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.034) 0:04:56.670 ***** 2021-11-20T07:28:37.604 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:37.611 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T07:28:37.613 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.636 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.033) 0:04:56.703 ***** 2021-11-20T07:28:37.643 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T07:28:37.645 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.670 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.736 ***** 2021-11-20T07:28:37.675 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.699 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.030) 0:04:56.767 ***** 2021-11-20T07:28:37.706 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T07:28:37.707 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.729 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.030) 0:04:56.797 ***** 2021-11-20T07:28:37.736 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T07:28:37.737 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.764 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.034) 0:04:56.832 ***** 2021-11-20T07:28:37.771 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T07:28:37.773 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.796 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.864 ***** 2021-11-20T07:28:37.803 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T07:28:37.805 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.828 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.031) 0:04:56.896 ***** 2021-11-20T07:28:37.836 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T07:28:37.838 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.860 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.031) 0:04:56.928 ***** 2021-11-20T07:28:37.866 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T07:28:37.868 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.892 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.032) 0:04:56.960 ***** 2021-11-20T07:28:37.899 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T07:28:37.901 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.030) 0:04:56.991 ***** 2021-11-20T07:28:37.929 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T07:28:37.931 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.962 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.036) 0:04:57.028 ***** 2021-11-20T07:28:37.968 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T07:28:37.969 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:37.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:37 +0000 (0:00:00.033) 0:04:57.061 ***** 2021-11-20T07:28:38.000 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T07:28:38.002 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.025 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.031) 0:04:57.093 ***** 2021-11-20T07:28:38.032 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T07:28:38.034 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.056 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.031) 0:04:57.124 ***** 2021-11-20T07:28:38.067 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T07:28:38.069 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.094 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.036) 0:04:57.160 ***** 2021-11-20T07:28:38.101 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T07:28:38.102 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.126 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:57.194 ***** 2021-11-20T07:28:38.133 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T07:28:38.134 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.162 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.036) 0:04:57.230 ***** 2021-11-20T07:28:38.171 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.194 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.031) 0:04:57.262 ***** 2021-11-20T07:28:38.202 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-11-20T07:28:38.203 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.229 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.035) 0:04:57.297 ***** 2021-11-20T07:28:38.240 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.264 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.034) 0:04:57.332 ***** 2021-11-20T07:28:38.271 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-20T07:28:38.273 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.298 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:57.365 ***** 2021-11-20T07:28:38.308 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.331 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:57.399 ***** 2021-11-20T07:28:38.338 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T07:28:38.340 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.364 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.032) 0:04:57.432 ***** 2021-11-20T07:28:38.373 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:38.375 INFO:teuthology.task.ansible.out:TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.399 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.034) 0:04:57.467 ***** 2021-11-20T07:28:38.405 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.432 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:57.500 ***** 2021-11-20T07:28:38.442 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T07:28:38.443 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.467 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.035) 0:04:57.535 ***** 2021-11-20T07:28:38.474 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-20T07:28:38.476 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.498 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.030) 0:04:57.566 ***** 2021-11-20T07:28:38.506 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-20T07:28:38.508 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.032) 0:04:57.598 ***** 2021-11-20T07:28:38.537 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T07:28:38.538 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.565 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.035) 0:04:57.633 ***** 2021-11-20T07:28:38.574 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.598 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.032) 0:04:57.666 ***** 2021-11-20T07:28:38.604 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-20T07:28:38.606 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.780 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.182) 0:04:57.848 ***** 2021-11-20T07:28:38.798 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T07:28:38.800 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.821 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.040) 0:04:57.889 ***** 2021-11-20T07:28:38.827 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T07:28:38.828 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.853 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.031) 0:04:57.920 ***** 2021-11-20T07:28:38.859 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-20T07:28:38.861 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.882 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.030) 0:04:57.950 ***** 2021-11-20T07:28:38.889 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T07:28:38.891 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.913 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.030) 0:04:57.981 ***** 2021-11-20T07:28:38.920 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T07:28:38.922 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:38.946 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:58.014 ***** 2021-11-20T07:28:38.954 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:38.956 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T07:28:38.980 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:38 +0000 (0:00:00.033) 0:04:58.048 ***** 2021-11-20T07:28:38.987 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T07:28:38.989 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.012 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.031) 0:04:58.080 ***** 2021-11-20T07:28:39.030 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T07:28:39.031 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.042 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T07:28:39.043 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} TASK [testnode : set_fact] ***************************************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi071.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.071 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.058) 0:04:58.139 ***** 2021-11-20T07:28:39.078 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.080 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.108 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.037) 0:04:58.176 ***** 2021-11-20T07:28:39.117 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.119 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T07:28:39.129 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.141 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.154 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.181 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.072) 0:04:58.249 ***** 2021-11-20T07:28:39.321 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003335", "end": "2021-11-20 07:28:39.292075", "rc": 0, "start": "2021-11-20 07:28:39.288740", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T07:28:39.349 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.167) 0:04:58.416 ***** 2021-11-20T07:28:39.380 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.406 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.057) 0:04:58.474 ***** 2021-11-20T07:28:39.415 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T07:28:39.417 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009812", "end": "2021-11-20 07:28:39.365868", "rc": 0, "start": "2021-11-20 07:28:39.356056", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T07:28:39.444 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.038) 0:04:58.512 ***** 2021-11-20T07:28:39.452 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T07:28:39.454 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.481 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.036) 0:04:58.549 ***** 2021-11-20T07:28:39.492 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.525 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.044) 0:04:58.593 ***** 2021-11-20T07:28:39.535 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.561 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.035) 0:04:58.628 ***** 2021-11-20T07:28:39.582 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.604 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.043) 0:04:58.672 ***** 2021-11-20T07:28:39.615 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.637 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.033) 0:04:58.705 ***** 2021-11-20T07:28:39.650 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.672 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.033) 0:04:58.739 ***** 2021-11-20T07:28:39.697 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-20T07:28:39.700 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:39.724 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:39 +0000 (0:00:00.052) 0:04:58.792 ***** 2021-11-20T07:28:41.656 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-20T07:28:41.658 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:28:41.681 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:41 +0000 (0:00:01.957) 0:05:00.749 ***** 2021-11-20T07:28:41.734 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140192768, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085008, 'block_used': 13010100, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085008, "block_size": 4096, "block_total": 240095108, "block_used": 13010100, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140192768, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T07:28:41.759 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:41 +0000 (0:00:00.077) 0:05:00.827 ***** 2021-11-20T07:28:41.767 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-20T07:28:41.769 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:28:41.795 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:41 +0000 (0:00:00.035) 0:05:00.863 ***** 2021-11-20T07:28:41.847 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T07:28:41.848 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140172288, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085003, 'block_used': 13010105, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085003, "block_size": 4096, "block_total": 240095108, "block_used": 13010105, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140172288, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T07:28:41.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:41 +0000 (0:00:00.079) 0:05:00.942 ***** 2021-11-20T07:28:41.947 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-20T07:28:41.949 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005159", "end": "2021-11-20 07:28:41.928185", "rc": 0, "start": "2021-11-20 07:28:41.923026", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:41.972 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:41 +0000 (0:00:00.097) 0:05:01.040 ***** 2021-11-20T07:28:42.013 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140192768, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085008, 'block_used': 13010100, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085008, "block_size": 4096, "block_total": 240095108, "block_used": 13010100, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140192768, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:42.037 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:42 +0000 (0:00:00.064) 0:05:01.105 ***** 2021-11-20T07:28:42.066 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T07:28:42.068 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005214", "end": "2021-11-20 07:28:42.048249", "rc": 0, "start": "2021-11-20 07:28:42.043035", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:42.090 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:42 +0000 (0:00:00.053) 0:05:01.158 ***** 2021-11-20T07:28:42.117 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:42 +0000 (0:00:00.026) 0:05:01.185 ***** 2021-11-20T07:28:42.169 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T07:28:42.171 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140172288, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085003, 'block_used': 13010105, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085003, "block_size": 4096, "block_total": 240095108, "block_used": 13010105, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140172288, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:42.194 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:42 +0000 (0:00:00.077) 0:05:01.262 ***** 2021-11-20T07:28:42.223 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T07:28:42.248 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:42 +0000 (0:00:00.054) 0:05:01.316 ***** 2021-11-20T07:28:43.887 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T07:28:43.888 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ5RgA6ZWSAYtCLsrUj1ucnScPpRxf818h'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.564926", "end": "2021-11-20 07:28:43.853405", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ5RgA6ZWSAYtCLsrUj1ucnScPpRxf818h"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:42.288479", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:44.159 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPjfSEwyS6O7reznTntmErFFPaCB3JEFa'], 'uuids': ['a0d2f707-d89e-4fbe-99db-6ef6bdf2c7ed'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.692362", "end": "2021-11-20 07:28:44.131594", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPjfSEwyS6O7reznTntmErFFPaCB3JEFa"], "labels": [], "masters": [], "uuids": ["a0d2f707-d89e-4fbe-99db-6ef6bdf2c7ed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:42.439232", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:45.150 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-1CXebH-9RT6-vsRV-HzJn-2lSQ-2elS-DzYAf2', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', 'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.072016", "end": "2021-11-20 07:28:45.119572", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-1CXebH-9RT6-vsRV-HzJn-2lSQ-2elS-DzYAf2", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:44.047556", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T07:28:45.152 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:45.409 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-adkxcd-5W6D-8sZ9-1vdt-xBI7-1kDK-ooeuoB', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN', 'nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.046230", "end": "2021-11-20 07:28:45.370257", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-adkxcd-5W6D-8sZ9-1vdt-xBI7-1kDK-ooeuoB", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN", "nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:44.324027", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T07:28:45.411 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:46.419 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ4EchxpPFfVUFqPUqPaz0ZNKMRG6lHkZY'], 'uuids': ['4386e38a-8493-4535-82c8-6c337c6cb4ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.068057", "end": "2021-11-20 07:28:46.376153", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ4EchxpPFfVUFqPUqPaz0ZNKMRG6lHkZY"], "labels": [], "masters": [], "uuids": ["4386e38a-8493-4535-82c8-6c337c6cb4ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:45.308096", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:46.672 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqbRZ5a1aR3PXD8CrvDjCOYH7Hbd2fnQMR'], 'uuids': ['72ed472a-f709-4f39-8559-884ba5ce1bbe'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.064578", "end": "2021-11-20 07:28:46.636538", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqbRZ5a1aR3PXD8CrvDjCOYH7Hbd2fnQMR"], "labels": [], "masters": [], "uuids": ["72ed472a-f709-4f39-8559-884ba5ce1bbe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:45.571960", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:47.687 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ8VLq4B1I6fKqJkJ762CXfGJDQh4aDqXg'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.065961", "end": "2021-11-20 07:28:47.651219", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ8VLq4B1I6fKqJkJ762CXfGJDQh4aDqXg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:46.585258", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:47.907 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxq6hh7x3YeTeNs0mqBVvDOKZ2bhQwyIMz5'], 'uuids': ['13d7d38b-2c9c-4dab-b67c-3f099fd800fc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.056665", "end": "2021-11-20 07:28:47.880004", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxq6hh7x3YeTeNs0mqBVvDOKZ2bhQwyIMz5"], "labels": [], "masters": [], "uuids": ["13d7d38b-2c9c-4dab-b67c-3f099fd800fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:46.823339", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:48.942 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZYmlD8wJ0ttwxzD1B8F3WXPUUWQgPTiDP'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.070819", "end": "2021-11-20 07:28:48.910369", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZYmlD8wJ0ttwxzD1B8F3WXPUUWQgPTiDP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:47.839550", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:48.953 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-35000c50091e19c4b', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR', 'wwn-0x5000c50091e19c4b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QJJR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e19c4b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-35000c50091e19c4b-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'wwn-0x5000c50091e19c4b-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-35000c50091e19c4b", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T07:28:48.955 INFO:teuthology.task.ansible.out:Z1W5QJJR-part1", "scsi-35000c50091e19c4b-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:49.155 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqfeGQIFRmUcHQntR76FjZB01WacTj3A0k'], 'uuids': ['f9c3b2ab-e661-4b8f-b258-4745a9618d65'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050073", "end": "2021-11-20 07:28:49.121128", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqfeGQIFRmUcHQntR76FjZB01WacTj3A0k"], "labels": [], "masters": [], "uuids": ["f9c3b2ab-e661-4b8f-b258-4745a9618d65"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:48.071055", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:49.168 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS', 'scsi-35000c50091e2ee9d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS', 'wwn-0x5000c50091e2ee9d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QZGS', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2ee9d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'scsi-35000c50091e2ee9d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS-part1', 'wwn-0x5000c50091e2ee9d-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS", "scsi-35000c50091e2ee9d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS", "wwn-0x5000c50091e2ee9d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T07:28:49.170 INFO:teuthology.task.ansible.out:Z1W5QZGS-part1", "scsi-35000c50091e2ee9d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS-part1", "wwn-0x5000c50091e2ee9d-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QZGS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2ee9d"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:50.222 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZCNrn8SVTsgbUhxgg6a0PWe9SN8rX5QAZ'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.068490", "end": "2021-11-20 07:28:50.184981", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZCNrn8SVTsgbUhxgg6a0PWe9SN8rX5QAZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:49.116491", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:50.254 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:50 +0000 (0:00:08.005) 0:05:09.322 ***** 2021-11-20T07:28:50.424 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPCXHUEWHHPL8xkG16psap5fi0UCwe3ce'], 'uuids': ['6fecec99-7e41-46bb-b849-4e1f848efdd7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.078193", "end": "2021-11-20 07:28:50.390516", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPCXHUEWHHPL8xkG16psap5fi0UCwe3ce"], "labels": [], "masters": [], "uuids": ["6fecec99-7e41-46bb-b849-4e1f848efdd7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:49.312323", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T07:28:50.468 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:50 +0000 (0:00:00.213) 0:05:09.535 ***** 2021-11-20T07:28:50.670 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T07:28:50.671 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ5RgA6ZWSAYtCLsrUj1ucnScPpRxf818h'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.145975", "end": "2021-11-20 07:28:50.627483", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ5RgA6ZWSAYtCLsrUj1ucnScPpRxf818h"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:50.481508", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143017 s, 806 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143017 s, 806 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:50.844 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPjfSEwyS6O7reznTntmErFFPaCB3JEFa'], 'uuids': ['a0d2f707-d89e-4fbe-99db-6ef6bdf2c7ed'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.143498", "end": "2021-11-20 07:28:50.813221", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPjfSEwyS6O7reznTntmErFFPaCB3JEFa"], "labels": [], "masters": [], "uuids": ["a0d2f707-d89e-4fbe-99db-6ef6bdf2c7ed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:50.669723", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140716 s, 820 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140716 s, 820 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:50.877 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-1CXebH-9RT6-vsRV-HzJn-2lSQ-2elS-DzYAf2', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN', 'nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039741", "end": "2021-11-20 07:28:50.846257", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-1CXebH-9RT6-vsRV-HzJn-2lSQ-2elS-DzYAf2", "nvme-INTEL_SSDPEDMD400G4_PHFT62040172400BGN", "nvme-nvme.8086-50484654363230343031373234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:50.806516", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0370391 s, 3.1 GB/s", "std 2021-11-20T07:28:50.879 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0370391 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.055 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-adkxcd-5W6D-8sZ9-1vdt-xBI7-1kDK-ooeuoB', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN', 'nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039383", "end": "2021-11-20 07:28:51.029754", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-adkxcd-5W6D-8sZ9-1vdt-xBI7-1kDK-ooeuoB", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN", "nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:50.990371", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0365794 s, 3.2 GB/s", "std 2021-11-20T07:28:51.057 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0365794 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.205 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ4EchxpPFfVUFqPUqPaz0ZNKMRG6lHkZY'], 'uuids': ['4386e38a-8493-4535-82c8-6c337c6cb4ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145153", "end": "2021-11-20 07:28:51.175667", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ4EchxpPFfVUFqPUqPaz0ZNKMRG6lHkZY"], "labels": [], "masters": [], "uuids": ["4386e38a-8493-4535-82c8-6c337c6cb4ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.030514", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142519 s, 809 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142519 s, 809 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.386 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqbRZ5a1aR3PXD8CrvDjCOYH7Hbd2fnQMR'], 'uuids': ['72ed472a-f709-4f39-8559-884ba5ce1bbe'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145850", "end": "2021-11-20 07:28:51.363723", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqbRZ5a1aR3PXD8CrvDjCOYH7Hbd2fnQMR"], "labels": [], "masters": [], "uuids": ["72ed472a-f709-4f39-8559-884ba5ce1bbe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.217873", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143192 s, 806 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143192 s, 806 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.523 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ8VLq4B1I6fKqJkJ762CXfGJDQh4aDqXg'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141343", "end": "2021-11-20 07:28:51.501869", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZ8VLq4B1I6fKqJkJ762CXfGJDQh4aDqXg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.360526", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138861 s, 831 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138861 s, 831 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.701 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxq6hh7x3YeTeNs0mqBVvDOKZ2bhQwyIMz5'], 'uuids': ['13d7d38b-2c9c-4dab-b67c-3f099fd800fc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142393", "end": "2021-11-20 07:28:51.677541", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxq6hh7x3YeTeNs0mqBVvDOKZ2bhQwyIMz5"], "labels": [], "masters": [], "uuids": ["13d7d38b-2c9c-4dab-b67c-3f099fd800fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.535148", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139706 s, 826 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139706 s, 826 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.869 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZYmlD8wJ0ttwxzD1B8F3WXPUUWQgPTiDP'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.157175", "end": "2021-11-20 07:28:51.841499", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZYmlD8wJ0ttwxzD1B8F3WXPUUWQgPTiDP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.684324", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154463 s, 747 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154463 s, 747 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:51.881 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR', 'scsi-35000c50091e19c4b', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR', 'wwn-0x5000c50091e19c4b'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QJJR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e19c4b', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR-part1', 'scsi-35000c50091e19c4b-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1', 'wwn-0x5000c50091e19c4b-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QJJR", "scsi-35000c50091e19c4b", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR", "wwn-0x5000c50091e19c4b"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QJJR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T07:28:51.884 INFO:teuthology.task.ansible.out:Z1W5QJJR-part1", "scsi-35000c50091e19c4b-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QJJR-part1", "wwn-0x5000c50091e19c4b-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QJJR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e19c4b"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:52.021 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqfeGQIFRmUcHQntR76FjZB01WacTj3A0k'], 'uuids': ['f9c3b2ab-e661-4b8f-b258-4745a9618d65'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.146449", "end": "2021-11-20 07:28:51.996287", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqfeGQIFRmUcHQntR76FjZB01WacTj3A0k"], "labels": [], "masters": [], "uuids": ["f9c3b2ab-e661-4b8f-b258-4745a9618d65"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:51.849838", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143786 s, 802 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143786 s, 802 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.033 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS', 'scsi-35000c50091e2ee9d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS', 'wwn-0x5000c50091e2ee9d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QZGS', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2ee9d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'scsi-35000c50091e2ee9d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS-part1', 'wwn-0x5000c50091e2ee9d-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QZGS", "scsi-35000c50091e2ee9d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS", "wwn-0x5000c50091e2ee9d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QZGS-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T07:28:52.035 INFO:teuthology.task.ansible.out:Z1W5QZGS-part1", "scsi-35000c50091e2ee9d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QZGS-part1", "wwn-0x5000c50091e2ee9d-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QZGS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2ee9d"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:52.204 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZCNrn8SVTsgbUhxgg6a0PWe9SN8rX5QAZ'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145636", "end": "2021-11-20 07:28:52.166553", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-I06DoDDkpDxyhbqfczWZWXnODsK76lrZCNrn8SVTsgbUhxgg6a0PWe9SN8rX5QAZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:52.020917", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143016 s, 807 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143016 s, 807 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.363 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:01.766) 0:05:11.302 ***** changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPCXHUEWHHPL8xkG16psap5fi0UCwe3ce'], 'uuids': ['6fecec99-7e41-46bb-b849-4e1f848efdd7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.144881", "end": "2021-11-20 07:28:52.320005", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-I3FdnwKxuuaR9QDtebzH8BhAYMzwaRxqPCXHUEWHHPL8xkG16psap5fi0UCwe3ce"], "labels": [], "masters": [], "uuids": ["6fecec99-7e41-46bb-b849-4e1f848efdd7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 07:28:52.175124", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142248 s, 811 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142248 s, 811 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.380 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.146) 0:05:11.448 ***** 2021-11-20T07:28:52.382 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:52.496 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T07:28:52.497 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.069074", "end": "2021-11-20 07:28:52.462657", "rc": 0, "start": "2021-11-20 07:28:52.393583", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.520 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.139) 0:05:11.588 ***** 2021-11-20T07:28:52.636 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.058040", "end": "2021-11-20 07:28:52.603873", "rc": 0, "start": "2021-11-20 07:28:52.545833", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.663 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.142) 0:05:11.731 ***** 2021-11-20T07:28:52.776 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T07:28:52.778 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.066417", "end": "2021-11-20 07:28:52.752718", "rc": 0, "start": "2021-11-20 07:28:52.686301", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.801 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.138) 0:05:11.869 ***** 2021-11-20T07:28:52.830 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T07:28:52.856 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.054) 0:05:11.924 ***** 2021-11-20T07:28:52.894 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:52.896 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.044422", "end": "2021-11-20 07:28:52.869942", "rc": 0, "start": "2021-11-20 07:28:52.825520", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:28:52.920 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.063) 0:05:11.988 ***** 2021-11-20T07:28:52.944 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.023) 0:05:12.011 ***** 2021-11-20T07:28:52.960 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T07:28:52.962 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:52.982 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:52 +0000 (0:00:00.038) 0:05:12.050 ***** 2021-11-20T07:28:52.990 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T07:28:53.016 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.033) 0:05:12.084 ***** 2021-11-20T07:28:53.022 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T07:28:53.025 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.051 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.035) 0:05:12.119 ***** 2021-11-20T07:28:53.058 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T07:28:53.060 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.086 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.034) 0:05:12.154 ***** 2021-11-20T07:28:53.092 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T07:28:53.094 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.116 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.030) 0:05:12.184 ***** 2021-11-20T07:28:53.123 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T07:28:53.125 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.148 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.032) 0:05:12.216 ***** 2021-11-20T07:28:53.176 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T07:28:53.177 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.201 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.053) 0:05:12.269 ***** 2021-11-20T07:28:53.240 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T07:28:53.242 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.060) 0:05:12.330 ***** 2021-11-20T07:28:53.300 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T07:28:53.302 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:28:53.325 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.062) 0:05:12.393 ***** 2021-11-20T07:28:53.364 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T07:28:53.365 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140192768, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085008, 'block_used': 13010100, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085008, "block_size": 4096, "block_total": 240095108, "block_used": 13010100, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140192768, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.387 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.061) 0:05:12.455 ***** 2021-11-20T07:28:53.415 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T07:28:53.417 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.439 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.051) 0:05:12.507 ***** 2021-11-20T07:28:53.456 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T07:28:53.458 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T07:28:53.481 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.040) 0:05:12.548 ***** 2021-11-20T07:28:53.486 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:53.488 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.031) 0:05:12.579 ***** 2021-11-20T07:28:53.513 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:53.534 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi112.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140172288, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085003, 'block_used': 13010105, 'inode_total': 61054976, 'inode_available': 60984947, 'inode_used': 70029, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085003, "block_size": 4096, "block_total": 240095108, "block_used": 13010105, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984947, "inode_total": 61054976, "inode_used": 70029, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140172288, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.555 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.043) 0:05:12.623 ***** 2021-11-20T07:28:53.562 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:53.564 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.588 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.033) 0:05:12.656 ***** 2021-11-20T07:28:53.595 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T07:28:53.597 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.622 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.034) 0:05:12.690 ***** 2021-11-20T07:28:53.629 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.653 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.030) 0:05:12.721 ***** 2021-11-20T07:28:53.660 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:28:53.662 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.686 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.032) 0:05:12.754 ***** 2021-11-20T07:28:53.693 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T07:28:53.695 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.717 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.031) 0:05:12.785 ***** 2021-11-20T07:28:53.726 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.750 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.032) 0:05:12.818 ***** 2021-11-20T07:28:53.756 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T07:28:53.758 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.781 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.031) 0:05:12.849 ***** 2021-11-20T07:28:53.788 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-20T07:28:53.790 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.814 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.032) 0:05:12.882 ***** 2021-11-20T07:28:53.842 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.865 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.051) 0:05:12.933 ***** 2021-11-20T07:28:53.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T07:28:53.896 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:53.917 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:53 +0000 (0:00:00.051) 0:05:12.985 ***** 2021-11-20T07:28:54.304 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-20T07:28:54.306 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T07:28:54.329 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:54 +0000 (0:00:00.411) 0:05:13.397 ***** 2021-11-20T07:28:54.369 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T07:28:54.394 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:54 +0000 (0:00:00.064) 0:05:13.462 ***** 2021-11-20T07:28:54.778 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T07:28:54.780 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:54.790 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.119 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.131 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.477 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.488 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.825 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:55.837 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:56.155 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:56.180 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:56 +0000 (0:00:01.785) 0:05:15.248 ***** 2021-11-20T07:28:56.197 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T07:28:56.227 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:56 +0000 (0:00:00.047) 0:05:15.295 ***** 2021-11-20T07:28:56.393 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T07:28:56.395 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T07:28:56.419 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:56 +0000 (0:00:00.192) 0:05:15.487 ***** 2021-11-20T07:28:56.428 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T07:28:56.455 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:56 +0000 (0:00:00.035) 0:05:15.523 ***** 2021-11-20T07:28:56.644 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T07:28:56.646 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:56.669 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:56.828 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:56.854 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:57.013 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:57.025 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:57.196 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:57.207 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T07:28:57.209 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.212 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.781) 0:05:16.305 ***** 2021-11-20T07:28:57.257 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.020) 0:05:16.325 ***** 2021-11-20T07:28:57.276 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-20T07:28:57.278 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.298 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.040) 0:05:16.366 ***** 2021-11-20T07:28:57.305 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.337 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.037) 0:05:16.404 ***** 2021-11-20T07:28:57.344 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.369 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.032) 0:05:16.437 ***** 2021-11-20T07:28:57.387 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:57.417 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.048) 0:05:16.485 ***** 2021-11-20T07:28:57.628 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-20T07:28:57.631 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:28:57.690 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.269) 0:05:16.755 ***** 2021-11-20T07:28:57.700 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:28:57.728 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.040) 0:05:16.796 ***** 2021-11-20T07:28:57.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T07:28:57.768 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.036) 0:05:16.832 ***** 2021-11-20T07:28:57.782 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T07:28:57.810 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:57 +0000 (0:00:00.045) 0:05:16.878 ***** 2021-11-20T07:28:58.160 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-11-20T07:28:58.162 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true} changed: [smithi071.front.sepia.ceph.com] => {"changed": true} 2021-11-20T07:28:58.187 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.377) 0:05:17.255 ***** 2021-11-20T07:28:58.208 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.021) 0:05:17.276 ***** 2021-11-20T07:28:58.583 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-20T07:28:58.586 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T07:28:58.606 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.398) 0:05:17.674 ***** 2021-11-20T07:28:58.614 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T07:28:58.639 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.032) 0:05:17.707 ***** 2021-11-20T07:28:58.646 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.672 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.033) 0:05:17.740 ***** 2021-11-20T07:28:58.674 INFO:teuthology.task.ansible.out: 2021-11-20T07:28:58.681 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.033) 0:05:17.773 ***** 2021-11-20T07:28:58.714 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.740 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.034) 0:05:17.808 ***** 2021-11-20T07:28:58.747 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.771 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.030) 0:05:17.839 ***** 2021-11-20T07:28:58.779 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T07:28:58.781 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.803 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.032) 0:05:17.871 ***** 2021-11-20T07:28:58.810 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.836 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.032) 0:05:17.904 ***** 2021-11-20T07:28:58.853 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.880 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.044) 0:05:17.948 ***** 2021-11-20T07:28:58.888 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.911 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.030) 0:05:17.978 ***** 2021-11-20T07:28:58.916 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-20T07:28:58.917 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.945 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.034) 0:05:18.013 ***** 2021-11-20T07:28:58.952 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:58.976 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:58 +0000 (0:00:00.031) 0:05:18.044 ***** 2021-11-20T07:28:58.984 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-11-20T07:28:58.986 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.008 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.031) 0:05:18.076 ***** 2021-11-20T07:28:59.015 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.038 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.030) 0:05:18.106 ***** 2021-11-20T07:28:59.048 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.034) 0:05:18.141 ***** 2021-11-20T07:28:59.082 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.106 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.032) 0:05:18.174 ***** 2021-11-20T07:28:59.113 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.147 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.040) 0:05:18.215 ***** 2021-11-20T07:28:59.155 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.182 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.035) 0:05:18.250 ***** 2021-11-20T07:28:59.189 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-20T07:28:59.191 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.217 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.034) 0:05:18.285 ***** 2021-11-20T07:28:59.226 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.252 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.033) 0:05:18.319 ***** 2021-11-20T07:28:59.259 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-11-20T07:28:59.261 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.285 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.033) 0:05:18.353 ***** 2021-11-20T07:28:59.293 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.318 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.032) 0:05:18.385 ***** 2021-11-20T07:28:59.325 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-20T07:28:59.327 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.351 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.033) 0:05:18.419 ***** 2021-11-20T07:28:59.361 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.383 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.032) 0:05:18.451 ***** 2021-11-20T07:28:59.390 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-20T07:28:59.392 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.415 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.031) 0:05:18.482 ***** 2021-11-20T07:28:59.422 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.446 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.031) 0:05:18.514 ***** 2021-11-20T07:28:59.453 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-20T07:28:59.455 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.032) 0:05:18.547 ***** 2021-11-20T07:28:59.486 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.510 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.031) 0:05:18.578 ***** 2021-11-20T07:28:59.517 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-20T07:28:59.519 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.543 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.032) 0:05:18.611 ***** 2021-11-20T07:28:59.552 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.577 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.033) 0:05:18.645 ***** 2021-11-20T07:28:59.585 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-20T07:28:59.586 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.610 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.033) 0:05:18.678 ***** 2021-11-20T07:28:59.620 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:28:59.645 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:28:59 +0000 (0:00:00.034) 0:05:18.713 ***** 2021-11-20T07:29:00.105 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-11-20T07:29:00.106 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T07:29:00.130 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.485) 0:05:19.198 ***** 2021-11-20T07:29:00.150 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T07:29:00.172 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.041) 0:05:19.240 ***** 2021-11-20T07:29:00.437 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-20T07:29:00.438 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 07:22:47 UTC", "ActiveEnterTimestampMonotonic": "12118109", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount sysinit.target -.mount ntpd.service sntp.service systemd-journald.socket basic.target ntpdate.service system.slice systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 07:22:46 UTC", "AssertTimestampMonotonic": "10508125", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 07:22:46 UTC", "ConditionTimestampMonotonic": "10508090", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-20T07:29:00.441 INFO:teuthology.task.ansible.out:emd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "900", "ExecMainStartTimestamp": "Sat 2021-11-20 07:22:47 UTC", "ExecMainStartTimestampMonotonic": "12054494", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 07:22:46 UTC", "InactiveExitTimestampMonotonic": "10597516", "InvocationID": "307b4a97aa5e4ea38551ef388cf38a81", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T07:29:00.442 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "900", "MemoryAccounting": "yes", "MemoryCurrent": "1601536", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount syste 2021-11-20T07:29:00.445 INFO:teuthology.task.ansible.out:m.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 07:22:47 UTC", "StateChangeTimestampMonotonic": "12118109", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 07:22:47 UTC", "WatchdogTimestampMonotonic": "12054593", "WatchdogUSec": "0"}} 2021-11-20T07:29:00.474 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.302) 0:05:19.542 ***** 2021-11-20T07:29:00.498 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 07:22:46 UTC", "ActiveEnterTimestampMonotonic": "12068281", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-tmpfiles-setup.service systemd-journald.socket -.mount system.slice ntpd.service sntp.service tmp.mount sysinit.target ntpdate.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 07:22:44 UTC", "AssertTimestampMonotonic": "10267558", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 07:22:44 UTC", "ConditionTimestampMonotonic": "10267530", "ConfigurationDirectoryMode": "0755", "Conflicts": "shut 2021-11-20T07:29:00.500 INFO:teuthology.task.ansible.out:down.target systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "843", "ExecMainStartTimestamp": "Sat 2021-11-20 07:22:46 UTC", "ExecMainStartTimestampMonotonic": "11939552", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 07:22:44 UTC", "InactiveExitTimestampMonotonic": "10271692", "InvocationID": "53df214c570046c2ac374c0d2f0b74b0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T07:29:00.502 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "843", "MemoryAccounting": "yes", "MemoryCurrent": "1912832", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target 2021-11-20T07:29:00.503 INFO:teuthology.task.ansible.out:-.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 07:22:46 UTC", "StateChangeTimestampMonotonic": "12068281", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 07:22:46 UTC", "WatchdogTimestampMonotonic": "11939588", "WatchdogUSec": "0"}} TASK [testnode : set_fact] ***************************************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T07:29:00.527 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.052) 0:05:19.595 ***** 2021-11-20T07:29:00.548 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.020) 0:05:19.615 ***** 2021-11-20T07:29:00.555 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T07:29:00.580 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.032) 0:05:19.648 ***** 2021-11-20T07:29:00.597 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:29:00.598 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T07:29:00.634 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.054) 0:05:19.702 ***** 2021-11-20T07:29:00.646 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T07:29:00.672 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.037) 0:05:19.740 ***** 2021-11-20T07:29:00.679 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-11-20T07:29:00.681 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.705 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.032) 0:05:19.773 ***** 2021-11-20T07:29:00.715 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.738 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.033) 0:05:19.806 ***** 2021-11-20T07:29:00.746 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-20T07:29:00.748 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.772 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.033) 0:05:19.840 ***** 2021-11-20T07:29:00.782 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.808 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.035) 0:05:19.876 ***** 2021-11-20T07:29:00.816 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.842 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.034) 0:05:19.910 ***** 2021-11-20T07:29:00.850 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:00.883 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:00 +0000 (0:00:00.035) 0:05:19.946 ***** 2021-11-20T07:29:01.046 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-20T07:29:01.048 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T07:29:01.070 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:01 +0000 (0:00:00.192) 0:05:20.138 ***** 2021-11-20T07:29:01.089 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T07:29:01.114 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:01 +0000 (0:00:00.043) 0:05:20.181 ***** 2021-11-20T07:29:01.568 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T07:29:01.599 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:01 +0000 (0:00:00.484) 0:05:20.666 ***** 2021-11-20T07:29:01.618 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T07:29:01.644 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:01 +0000 (0:00:00.045) 0:05:20.712 ***** 2021-11-20T07:29:02.724 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T07:29:02.726 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T07:29:02.749 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:01.105) 0:05:21.817 ***** 2021-11-20T07:29:02.770 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:00.020) 0:05:21.838 ***** 2021-11-20T07:29:02.777 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-20T07:29:02.779 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:02.802 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:00.031) 0:05:21.870 ***** 2021-11-20T07:29:02.809 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:02.834 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:00.031) 0:05:21.902 ***** 2021-11-20T07:29:02.842 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T07:29:02.866 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:00.032) 0:05:21.934 ***** 2021-11-20T07:29:02.874 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T07:29:02.899 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:02 +0000 (0:00:00.032) 0:05:21.967 ***** 2021-11-20T07:29:03.045 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-11-20T07:29:03.047 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T07:29:03.070 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.169) 0:05:22.137 ***** 2021-11-20T07:29:03.087 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T07:29:03.112 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.042) 0:05:22.179 ***** 2021-11-20T07:29:03.121 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.147 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.034) 0:05:22.214 ***** 2021-11-20T07:29:03.153 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.179 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.032) 0:05:22.247 ***** 2021-11-20T07:29:03.192 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T07:29:03.211 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.032) 0:05:22.279 ***** 2021-11-20T07:29:03.219 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T07:29:03.243 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.031) 0:05:22.311 ***** 2021-11-20T07:29:03.255 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.277 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.032) 0:05:22.344 ***** 2021-11-20T07:29:03.284 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.310 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.034) 0:05:22.378 ***** 2021-11-20T07:29:03.477 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T07:29:03.500 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.190) 0:05:22.568 ***** 2021-11-20T07:29:03.508 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T07:29:03.533 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.033) 0:05:22.601 ***** 2021-11-20T07:29:03.690 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-11-20T07:29:03.692 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T07:29:03.714 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.180) 0:05:22.782 ***** 2021-11-20T07:29:03.716 INFO:teuthology.task.ansible.out: 2021-11-20T07:29:03.735 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T07:29:03.759 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.044) 0:05:22.827 ***** 2021-11-20T07:29:03.766 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-20T07:29:03.767 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.790 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.031) 0:05:22.858 ***** 2021-11-20T07:29:03.798 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.828 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.037) 0:05:22.896 ***** 2021-11-20T07:29:03.849 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-20T07:29:03.851 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.862 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.874 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.875 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.904 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.076) 0:05:22.972 ***** 2021-11-20T07:29:03.913 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.914 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.942 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.037) 0:05:23.010 ***** 2021-11-20T07:29:03.951 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-20T07:29:03.952 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.964 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:03.968 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi071.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.000 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:03 +0000 (0:00:00.055) 0:05:23.066 ***** 2021-11-20T07:29:04.011 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.014 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.025 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.026 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T07:29:04.027 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.056 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.056) 0:05:23.122 ***** 2021-11-20T07:29:04.084 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.029) 0:05:23.152 ***** 2021-11-20T07:29:04.095 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.116 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.032) 0:05:23.184 ***** 2021-11-20T07:29:04.139 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.159 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.042) 0:05:23.227 ***** 2021-11-20T07:29:04.165 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-11-20T07:29:04.167 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.195 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.035) 0:05:23.262 ***** 2021-11-20T07:29:04.204 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-20T07:29:04.206 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.228 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.033) 0:05:23.296 ***** 2021-11-20T07:29:04.236 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T07:29:04.238 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.278 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.038) 0:05:23.334 ***** TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.303 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.036) 0:05:23.371 ***** 2021-11-20T07:29:04.310 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.339 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.035) 0:05:23.407 ***** 2021-11-20T07:29:04.346 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T07:29:04.348 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.370 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.031) 0:05:23.438 ***** 2021-11-20T07:29:04.376 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.403 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.032) 0:05:23.471 ***** 2021-11-20T07:29:04.411 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-20T07:29:04.413 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.439 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.036) 0:05:23.507 ***** 2021-11-20T07:29:04.448 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-20T07:29:04.450 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.477 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.038) 0:05:23.545 ***** 2021-11-20T07:29:04.486 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T07:29:04.488 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.514 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.036) 0:05:23.582 ***** 2021-11-20T07:29:04.521 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T07:29:04.523 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.548 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.032) 0:05:23.615 ***** 2021-11-20T07:29:04.556 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.585 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.037) 0:05:23.653 ***** 2021-11-20T07:29:04.613 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T07:29:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:04.636 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:04 +0000 (0:00:00.050) 0:05:23.704 ***** 2021-11-20T07:29:04.769 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-20T07:29:04.771 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005033", "end": "2021-11-20 07:29:04.731795", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 07:29:04.726762", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:04.854 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005189", "end": "2021-11-20 07:29:04.825130", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 07:29:04.819941", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:04.939 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003263", "end": "2021-11-20 07:29:04.911005", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 07:29:04.907742", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.034 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003006", "end": "2021-11-20 07:29:05.001218", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 07:29:04.998212", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.118 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002901", "end": "2021-11-20 07:29:05.085942", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 07:29:05.083041", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.212 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002875", "end": "2021-11-20 07:29:05.174980", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 07:29:05.172105", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.295 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003283", "end": "2021-11-20 07:29:05.259473", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 07:29:05.256190", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.320 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.684) 0:05:24.388 ***** 2021-11-20T07:29:05.348 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T07:29:05.370 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.050) 0:05:24.438 ***** 2021-11-20T07:29:05.388 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-20T07:29:05.391 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003125", "end": "2021-11-20 07:29:05.356229", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 07:29:05.353104", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:05.416 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.045) 0:05:24.484 ***** 2021-11-20T07:29:05.425 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-11-20T07:29:05.426 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:05.449 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T07:29:05.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.062) 0:05:24.547 ***** 2021-11-20T07:29:05.506 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:05.575 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-11-20T07:29:05.594 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.115) 0:05:24.662 ***** 2021-11-20T07:29:05.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:05 +0000 (0:00:00.022) 0:05:24.685 ***** 2021-11-20T07:29:06.250 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi071.front.sepia.ceph.com] 2021-11-20T07:29:06.289 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2021-11-20T07:29:06.316 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.698) 0:05:25.384 ***** 2021-11-20T07:29:06.365 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.049) 0:05:25.433 ***** 2021-11-20T07:29:06.372 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T07:29:06.374 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:06.398 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.033) 0:05:25.466 ***** 2021-11-20T07:29:06.407 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:06.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.035) 0:05:25.502 ***** 2021-11-20T07:29:06.443 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T07:29:06.445 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T07:29:06.470 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.036) 0:05:25.538 ***** 2021-11-20T07:29:06.479 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T07:29:06.505 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.034) 0:05:25.573 ***** 2021-11-20T07:29:06.523 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T07:29:06.525 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T07:29:06.550 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.044) 0:05:25.617 ***** 2021-11-20T07:29:06.558 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T07:29:06.585 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:06 +0000 (0:00:00.035) 0:05:25.653 ***** 2021-11-20T07:29:49.412 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T07:29:49.414 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T07:29:49.436 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:49 +0000 (0:00:42.851) 0:06:08.504 ***** 2021-11-20T07:29:49.475 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T07:29:49.477 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T07:29:49.497 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:49 +0000 (0:00:00.060) 0:06:08.565 ***** 2021-11-20T07:29:49.524 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:49.551 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:49 +0000 (0:00:00.053) 0:06:08.619 ***** 2021-11-20T07:29:49.777 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T07:29:49.779 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:29:49.924 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T07:29:49.926 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T07:29:49.948 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:49 +0000 (0:00:00.397) 0:06:09.016 ***** 2021-11-20T07:29:49.976 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T07:29:49.978 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:29:50.003 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:50 +0000 (0:00:00.054) 0:06:09.071 ***** 2021-11-20T07:29:50.011 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T07:29:50.012 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T07:29:50.036 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:50 +0000 (0:00:00.031) 0:06:09.103 ***** 2021-11-20T07:29:50.062 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:50.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:50 +0000 (0:00:00.050) 0:06:09.153 ***** 2021-11-20T07:29:50.307 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T07:29:50.308 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:29:50.518 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T07:29:50.546 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:50 +0000 (0:00:00.461) 0:06:09.614 ***** 2021-11-20T07:29:50.554 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-20T07:29:50.556 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393390.0696123-24708-155787684497506/source", "state": "file", "uid": 0} 2021-11-20T07:29:51.017 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393390.56163-24708-116384737491266/source", "state": "file", "uid": 0} 2021-11-20T07:29:51.042 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:51 +0000 (0:00:00.495) 0:06:10.110 ***** 2021-11-20T07:29:51.094 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393390.6168482-24796-209316826971880/source", "state": "file", "uid": 0} 2021-11-20T07:29:51.606 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637393391.1426911-24796-58801977867443/source", "state": "file", "uid": 0} 2021-11-20T07:29:51.639 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:51 +0000 (0:00:00.596) 0:06:10.706 ***** 2021-11-20T07:29:53.024 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-11-20T07:29:53.025 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:29:53.048 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:53 +0000 (0:00:01.409) 0:06:12.116 ***** 2021-11-20T07:29:53.657 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T07:29:53.692 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:53 +0000 (0:00:00.644) 0:06:12.760 ***** 2021-11-20T07:29:56.254 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6qimhnt2-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6qimhnt2-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl" 2021-11-20T07:29:56.256 INFO:teuthology.task.ansible.out:, "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T07:29:56.279 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:02.586) 0:06:15.347 ***** 2021-11-20T07:29:56.308 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T07:29:56.310 INFO:teuthology.task.ansible.out:skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:56.334 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.054) 0:06:15.402 ***** 2021-11-20T07:29:56.365 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:56.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.054) 0:06:15.456 ***** 2021-11-20T07:29:56.735 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T07:29:56.737 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.151071", "end": "2021-11-20 07:29:56.711182", "rc": 0, "start": "2021-11-20 07:29:56.560111", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:56.759 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.370) 0:06:15.827 ***** 2021-11-20T07:29:56.800 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi071.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:56.847 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T07:29:56.849 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-c3dnz0yf-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-c3dnz0yf-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T07:29:56.851 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T07:29:56.870 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.110) 0:06:15.938 ***** 2021-11-20T07:29:56.899 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T07:29:56.900 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:56.922 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.051) 0:06:15.990 ***** 2021-11-20T07:29:56.951 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:56.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:56 +0000 (0:00:00.051) 0:06:16.042 ***** 2021-11-20T07:29:57.327 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T07:29:57.329 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.149219", "end": "2021-11-20 07:29:57.300155", "rc": 0, "start": "2021-11-20 07:29:57.150936", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T07:29:57.354 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:57 +0000 (0:00:00.374) 0:06:16.416 ***** 2021-11-20T07:29:57.379 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T07:29:57.443 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-11-20T07:29:57.444 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-11-20T07:29:57.446 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-11-20T07:29:57.447 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T07:29:57.459 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 20 November 2021 07:29:57 +0000 (0:00:00.110) 0:06:16.527 ***** 2021-11-20T07:29:58.141 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2021-11-20T07:29:59.126 INFO:teuthology.task.ansible.out:ok: [smithi071.front.sepia.ceph.com] 2021-11-20T07:29:59.144 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-11-20T07:29:59.146 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:59 +0000 (0:00:01.684) 0:06:18.212 ***** 2021-11-20T07:29:59.353 INFO:teuthology.task.ansible.out:changed: [smithi071.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T07:29:59.385 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T07:29:59.387 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi071.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 smithi112.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 2021-11-20T07:29:59.387 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 07:29:59 +0000 (0:00:00.243) 0:06:18.455 ***** =============================================================================== 2021-11-20T07:29:59.389 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 60.98s container-host : Install container packages ---------------------------- 42.85s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.83s users : Update authorized_keys using the keys repo --------------------- 32.25s Remove revoked users --------------------------------------------------- 18.97s testnode : Ensure ceph packages are not present. ----------------------- 17.66s common : nrpe - Remove SELinux policy package -------------------------- 15.72s common : nrpe - Load SELinux policy package ---------------------------- 15.35s testnode : Install packages --------------------------------------------- 9.43s testnode : Upgrade packages --------------------------------------------- 8.38s testnode : Zap all non-root disks --------------------------------------- 8.01s testnode : Install GPG -------------------------------------------------- 7.16s common : Install Kerberos Packages (RedHat) ----------------------------- 4.47s testnode : Ensure ceph dependency packages are not present. ------------- 3.08s testnode : Install GPG keys --------------------------------------------- 2.64s container-host : Install registries-conf-ctl ---------------------------- 2.59s testnode : Rebuild rpmdb ------------------------------------------------ 2.56s testnode : Make sure rpm dependencies are installed --------------------- 1.96s testnode : Remove packages ---------------------------------------------- 1.91s testnode : Create logical volume(s) ------------------------------------- 1.79s 2021-11-20T07:29:59.848 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2021-11-20T07:29:59.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2021-11-20T07:29:59.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:29:59.923 DEBUG:teuthology.orchestra.run.smithi071:> true 2021-11-20T07:29:59.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2021-11-20T07:29:59.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T07:30:00.062 DEBUG:teuthology.orchestra.run.smithi112:> true 2021-11-20T07:30:00.127 INFO:teuthology.run_tasks:Running task clock... 2021-11-20T07:30:00.142 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-11-20T07:30:00.143 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T07:30:00.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T07:30:00.148 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T07:30:00.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T07:30:00.176 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T07:30:00.190 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T07:30:00.199 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T07:30:00.213 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T07:30:00.230 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2021-11-20T07:30:00.240 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2021-11-20T07:30:00.247 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2021-11-20T07:30:00.253 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T07:30:00.256 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2021-11-20T07:30:00.267 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T07:30:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T07:30:00.283 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T07:30:00.333 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2021-11-20T07:30:00.336 INFO:teuthology.orchestra.run.smithi071.stdout:210 Number of sources = 4 2021-11-20T07:30:00.336 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T07:30:00.336 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2021-11-20T07:30:00.337 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.337 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.337 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.337 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.354 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2021-11-20T07:30:00.356 INFO:teuthology.orchestra.run.smithi112.stdout:210 Number of sources = 4 2021-11-20T07:30:00.356 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T07:30:00.357 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2021-11-20T07:30:00.357 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.357 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.357 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.357 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T07:30:00.358 INFO:teuthology.run_tasks:Running task pexec... 2021-11-20T07:30:00.370 INFO:teuthology.task.pexec:Executing custom commands... 2021-11-20T07:30:00.371 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:30:00.371 DEBUG:teuthology.orchestra.run.smithi071:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T07:30:00.372 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi112.front.sepia.ceph.com 2021-11-20T07:30:00.373 DEBUG:teuthology.orchestra.run.smithi112:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T07:30:00.943 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 07:28:05 AM UTC. 2021-11-20T07:30:00.968 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 07:28:05 AM UTC. 2021-11-20T07:30:02.351 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:30:02.352 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:02.352 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:30:02.353 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:02.353 INFO:teuthology.orchestra.run.smithi071.stdout:Resetting modules: 2021-11-20T07:30:02.353 INFO:teuthology.orchestra.run.smithi071.stdout: container-tools 2021-11-20T07:30:02.353 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:02.354 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:30:02.354 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:02.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:02.380 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:30:02.381 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:02.381 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:30:02.381 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:02.382 INFO:teuthology.orchestra.run.smithi112.stdout:Resetting modules: 2021-11-20T07:30:02.382 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools 2021-11-20T07:30:02.382 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:02.382 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:30:02.383 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:02.383 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:02.576 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:30:02.610 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:30:03.180 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Nov 2021 07:28:05 AM UTC. 2021-11-20T07:30:03.232 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Nov 2021 07:28:05 AM UTC. 2021-11-20T07:30:04.586 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:30:04.589 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:30:04.589 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:30:04.590 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:30:04.590 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2021-11-20T07:30:04.590 INFO:teuthology.orchestra.run.smithi071.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T07:30:04.590 INFO:teuthology.orchestra.run.smithi071.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T07:30:04.591 INFO:teuthology.orchestra.run.smithi071.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T07:30:04.591 INFO:teuthology.orchestra.run.smithi071.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T07:30:04.591 INFO:teuthology.orchestra.run.smithi071.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T07:30:04.591 INFO:teuthology.orchestra.run.smithi071.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T07:30:04.591 INFO:teuthology.orchestra.run.smithi071.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T07:30:04.592 INFO:teuthology.orchestra.run.smithi071.stdout:Installing group/module packages: 2021-11-20T07:30:04.592 INFO:teuthology.orchestra.run.smithi071.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T07:30:04.592 INFO:teuthology.orchestra.run.smithi071.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T07:30:04.592 INFO:teuthology.orchestra.run.smithi071.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T07:30:04.593 INFO:teuthology.orchestra.run.smithi071.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T07:30:04.593 INFO:teuthology.orchestra.run.smithi071.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T07:30:04.593 INFO:teuthology.orchestra.run.smithi071.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T07:30:04.593 INFO:teuthology.orchestra.run.smithi071.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T07:30:04.593 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:30:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T07:30:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T07:30:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T07:30:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T07:30:04.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T07:30:04.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T07:30:04.595 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T07:30:04.595 INFO:teuthology.orchestra.run.smithi071.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T07:30:04.595 INFO:teuthology.orchestra.run.smithi071.stdout:Installing module profiles: 2021-11-20T07:30:04.596 INFO:teuthology.orchestra.run.smithi071.stdout: container-tools/common 2021-11-20T07:30:04.596 INFO:teuthology.orchestra.run.smithi071.stdout:Enabling module streams: 2021-11-20T07:30:04.596 INFO:teuthology.orchestra.run.smithi071.stdout: container-tools rhel8 2021-11-20T07:30:04.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:04.597 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:30:04.597 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:30:04.597 INFO:teuthology.orchestra.run.smithi071.stdout:Install 15 Packages 2021-11-20T07:30:04.597 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 7 Packages 2021-11-20T07:30:04.598 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:04.599 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 41 M 2021-11-20T07:30:04.599 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:30:04.665 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:30:04.667 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:30:04.668 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:30:04.668 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:30:04.668 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2021-11-20T07:30:04.668 INFO:teuthology.orchestra.run.smithi112.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T07:30:04.669 INFO:teuthology.orchestra.run.smithi112.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T07:30:04.669 INFO:teuthology.orchestra.run.smithi112.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T07:30:04.669 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T07:30:04.669 INFO:teuthology.orchestra.run.smithi112.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T07:30:04.670 INFO:teuthology.orchestra.run.smithi112.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T07:30:04.670 INFO:teuthology.orchestra.run.smithi112.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T07:30:04.670 INFO:teuthology.orchestra.run.smithi112.stdout:Installing group/module packages: 2021-11-20T07:30:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T07:30:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T07:30:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T07:30:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T07:30:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T07:30:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T07:30:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T07:30:04.675 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:30:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T07:30:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T07:30:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T07:30:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T07:30:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T07:30:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T07:30:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T07:30:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T07:30:04.677 INFO:teuthology.orchestra.run.smithi112.stdout:Installing module profiles: 2021-11-20T07:30:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools/common 2021-11-20T07:30:04.677 INFO:teuthology.orchestra.run.smithi112.stdout:Enabling module streams: 2021-11-20T07:30:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools rhel8 2021-11-20T07:30:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:04.678 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:30:04.678 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:30:04.678 INFO:teuthology.orchestra.run.smithi112.stdout:Install 15 Packages 2021-11-20T07:30:04.678 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 7 Packages 2021-11-20T07:30:04.679 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:04.679 INFO:teuthology.orchestra.run.smithi071.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 2.4 MB/s | 193 kB 00:00 2021-11-20T07:30:04.682 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 41 M 2021-11-20T07:30:04.682 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:30:04.683 INFO:teuthology.orchestra.run.smithi071.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 5.0 MB/s | 457 kB 00:00 2021-11-20T07:30:04.688 INFO:teuthology.orchestra.run.smithi071.stdout:(3/22): python3-podman-3.2.1-4.module_el8.6.0+9 12 MB/s | 148 kB 00:00 2021-11-20T07:30:04.688 INFO:teuthology.orchestra.run.smithi071.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 8.3 MB/s | 25 kB 00:00 2021-11-20T07:30:04.697 INFO:teuthology.orchestra.run.smithi071.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 8.4 MB/s | 94 kB 00:00 2021-11-20T07:30:04.777 INFO:teuthology.orchestra.run.smithi112.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 1.8 MB/s | 193 kB 00:00 2021-11-20T07:30:04.805 INFO:teuthology.orchestra.run.smithi112.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 3.4 MB/s | 457 kB 00:00 2021-11-20T07:30:04.806 INFO:teuthology.orchestra.run.smithi112.stdout:(3/22): python3-podman-3.2.1-4.module_el8.6.0+9 5.0 MB/s | 148 kB 00:00 2021-11-20T07:30:04.807 INFO:teuthology.orchestra.run.smithi112.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 12 MB/s | 25 kB 00:00 2021-11-20T07:30:04.809 INFO:teuthology.orchestra.run.smithi112.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 36 MB/s | 94 kB 00:00 2021-11-20T07:30:04.814 INFO:teuthology.orchestra.run.smithi071.stdout:(6/22): buildah-1.23.1-2.module_el8.6.0+954+963 36 MB/s | 7.9 MB 00:00 2021-11-20T07:30:04.822 INFO:teuthology.orchestra.run.smithi071.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 5.5 MB/s | 48 kB 00:00 2021-11-20T07:30:04.838 INFO:teuthology.orchestra.run.smithi071.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 2.6 MB/s | 41 kB 00:00 2021-11-20T07:30:04.846 INFO:teuthology.orchestra.run.smithi071.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 23 MB/s | 195 kB 00:00 2021-11-20T07:30:04.856 INFO:teuthology.orchestra.run.smithi071.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 12 MB/s | 97 kB 00:00 2021-11-20T07:30:04.870 INFO:teuthology.orchestra.run.smithi071.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 2.2 MB/s | 34 kB 00:00 2021-11-20T07:30:04.880 INFO:teuthology.orchestra.run.smithi071.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 12 MB/s | 123 kB 00:00 2021-11-20T07:30:04.889 INFO:teuthology.orchestra.run.smithi071.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 20 MB/s | 177 kB 00:00 2021-11-20T07:30:04.897 INFO:teuthology.orchestra.run.smithi071.stdout:(14/22): conmon-2.0.30-1.module_el8.6.0+944+d41 7.2 MB/s | 54 kB 00:00 2021-11-20T07:30:05.003 INFO:teuthology.orchestra.run.smithi071.stdout:(15/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 21 MB/s | 6.7 MB 00:00 2021-11-20T07:30:05.010 INFO:teuthology.orchestra.run.smithi071.stdout:(16/22): toolbox-0.0.99.3-1.module_el8.5.0+877+ 7.2 MB/s | 2.3 MB 00:00 2021-11-20T07:30:05.021 INFO:teuthology.orchestra.run.smithi071.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 6.6 MB/s | 73 kB 00:00 2021-11-20T07:30:05.024 INFO:teuthology.orchestra.run.smithi071.stdout:(18/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 24 MB/s | 518 kB 00:00 2021-11-20T07:30:05.029 INFO:teuthology.orchestra.run.smithi071.stdout:(19/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 8.2 MB/s | 70 kB 00:00 2021-11-20T07:30:05.046 INFO:teuthology.orchestra.run.smithi071.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 3.1 MB/s | 51 kB 00:00 2021-11-20T07:30:05.090 INFO:teuthology.orchestra.run.smithi112.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 8.0 MB/s | 2.3 MB 00:00 2021-11-20T07:30:05.111 INFO:teuthology.orchestra.run.smithi112.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 2.3 MB/s | 48 kB 00:00 2021-11-20T07:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 1.6 MB/s | 41 kB 00:00 2021-11-20T07:30:05.179 INFO:teuthology.orchestra.run.smithi112.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 5.6 MB/s | 195 kB 00:00 2021-11-20T07:30:05.179 INFO:teuthology.orchestra.run.smithi071.stdout:(21/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 20 MB/s | 3.1 MB 00:00 2021-11-20T07:30:05.191 INFO:teuthology.orchestra.run.smithi112.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 6.5 MB/s | 97 kB 00:00 2021-11-20T07:30:05.199 INFO:teuthology.orchestra.run.smithi112.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 3.0 MB/s | 34 kB 00:00 2021-11-20T07:30:05.206 INFO:teuthology.orchestra.run.smithi112.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 14 MB/s | 123 kB 00:00 2021-11-20T07:30:05.229 INFO:teuthology.orchestra.run.smithi112.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 7.3 MB/s | 177 kB 00:00 2021-11-20T07:30:05.251 INFO:teuthology.orchestra.run.smithi112.stdout:(14/22): conmon-2.0.30-1.module_el8.6.0+944+d41 2.5 MB/s | 54 kB 00:00 2021-11-20T07:30:05.296 INFO:teuthology.orchestra.run.smithi112.stdout:(15/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 14 MB/s | 6.7 MB 00:00 2021-11-20T07:30:05.338 INFO:teuthology.orchestra.run.smithi112.stdout:(16/22): buildah-1.23.1-2.module_el8.6.0+954+96 12 MB/s | 7.9 MB 00:00 2021-11-20T07:30:05.341 INFO:teuthology.orchestra.run.smithi112.stdout:(17/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 11 MB/s | 518 kB 00:00 2021-11-20T07:30:05.348 INFO:teuthology.orchestra.run.smithi112.stdout:(18/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 7.1 MB/s | 73 kB 00:00 2021-11-20T07:30:05.349 INFO:teuthology.orchestra.run.smithi112.stdout:(19/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 8.0 MB/s | 70 kB 00:00 2021-11-20T07:30:05.360 INFO:teuthology.orchestra.run.smithi112.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 4.7 MB/s | 51 kB 00:00 2021-11-20T07:30:05.445 INFO:teuthology.orchestra.run.smithi112.stdout:(21/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 32 MB/s | 3.1 MB 00:00 2021-11-20T07:30:05.765 INFO:teuthology.orchestra.run.smithi071.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 22 MB/s | 19 MB 00:00 2021-11-20T07:30:05.769 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:30:05.770 INFO:teuthology.orchestra.run.smithi071.stdout:Total 35 MB/s | 41 MB 00:01 2021-11-20T07:30:05.803 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:30:05.804 INFO:teuthology.orchestra.run.smithi112.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 35 MB/s | 19 MB 00:00 2021-11-20T07:30:05.807 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:30:05.808 INFO:teuthology.orchestra.run.smithi112.stdout:Total 36 MB/s | 41 MB 00:01 2021-11-20T07:30:05.830 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:30:05.830 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:30:05.841 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:30:05.868 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:30:05.868 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:30:06.168 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:30:06.171 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:30:06.207 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:30:06.210 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:30:07.300 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:30:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:30:07.432 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T07:30:07.479 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T07:30:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T07:30:07.684 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T07:30:08.172 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T07:30:08.233 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T07:30:08.368 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T07:30:08.373 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T07:30:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T07:30:08.593 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T07:30:08.746 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T07:30:08.778 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T07:30:08.950 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T07:30:08.960 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T07:30:09.154 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T07:30:09.206 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T07:30:09.330 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T07:30:09.348 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T07:30:09.462 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T07:30:09.540 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T07:30:09.636 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T07:30:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T07:30:09.811 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T07:30:09.922 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T07:30:09.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T07:30:10.141 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T07:30:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T07:30:10.956 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T07:30:11.061 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T07:30:11.162 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T07:30:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T07:30:11.308 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T07:30:12.846 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T07:30:12.913 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T07:30:13.029 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T07:30:13.131 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T07:30:13.236 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T07:30:13.346 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T07:30:13.613 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T07:30:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T07:30:14.345 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T07:30:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T07:30:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T07:30:14.537 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T07:30:14.667 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T07:30:14.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T07:30:14.832 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T07:30:14.833 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T07:30:14.982 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T07:30:14.991 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T07:30:15.129 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T07:30:15.153 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T07:30:15.273 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T07:30:15.299 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T07:30:15.442 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T07:30:15.466 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T07:30:15.712 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T07:30:15.713 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T07:30:15.789 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T07:30:15.805 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T07:30:16.179 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T07:30:16.179 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T07:30:16.180 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T07:30:16.180 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T07:30:16.180 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T07:30:16.181 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T07:30:16.181 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T07:30:16.182 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T07:30:16.182 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T07:30:16.182 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T07:30:16.183 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T07:30:16.183 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T07:30:16.183 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T07:30:16.184 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T07:30:16.184 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T07:30:16.184 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T07:30:16.184 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T07:30:16.184 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T07:30:16.185 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T07:30:16.185 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T07:30:16.185 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T07:30:16.185 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T07:30:16.186 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T07:30:16.186 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T07:30:16.186 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T07:30:16.187 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T07:30:16.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T07:30:16.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T07:30:16.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T07:30:16.275 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T07:30:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T07:30:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T07:30:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T07:30:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T07:30:16.277 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T07:30:16.277 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T07:30:16.277 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T07:30:16.277 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T07:30:16.277 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T07:30:16.278 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T07:30:16.278 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T07:30:16.279 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T07:30:16.279 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T07:30:16.280 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T07:30:16.280 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T07:30:16.281 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T07:30:16.281 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T07:30:16.282 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T07:30:16.282 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T07:30:16.282 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T07:30:16.283 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T07:30:16.283 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T07:30:16.284 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T07:30:16.284 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T07:30:16.285 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T07:30:16.285 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T07:30:16.285 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T07:30:16.286 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T07:30:16.893 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T07:30:16.893 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:16.893 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2021-11-20T07:30:16.894 INFO:teuthology.orchestra.run.smithi071.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T07:30:16.894 INFO:teuthology.orchestra.run.smithi071.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T07:30:16.894 INFO:teuthology.orchestra.run.smithi071.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.894 INFO:teuthology.orchestra.run.smithi071.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.895 INFO:teuthology.orchestra.run.smithi071.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.895 INFO:teuthology.orchestra.run.smithi071.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.895 INFO:teuthology.orchestra.run.smithi071.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.895 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:30:16.895 INFO:teuthology.orchestra.run.smithi071.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:16.896 INFO:teuthology.orchestra.run.smithi071.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T07:30:16.896 INFO:teuthology.orchestra.run.smithi071.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:16.896 INFO:teuthology.orchestra.run.smithi071.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T07:30:16.896 INFO:teuthology.orchestra.run.smithi071.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T07:30:16.897 INFO:teuthology.orchestra.run.smithi071.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T07:30:16.897 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T07:30:16.897 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T07:30:16.897 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T07:30:16.897 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T07:30:16.898 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T07:30:16.898 INFO:teuthology.orchestra.run.smithi071.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:16.898 INFO:teuthology.orchestra.run.smithi071.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T07:30:16.898 INFO:teuthology.orchestra.run.smithi071.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T07:30:16.899 INFO:teuthology.orchestra.run.smithi071.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T07:30:16.899 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:16.899 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:30:16.990 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T07:30:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:16.991 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2021-11-20T07:30:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T07:30:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T07:30:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T07:30:16.993 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:30:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T07:30:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T07:30:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T07:30:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T07:30:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T07:30:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T07:30:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T07:30:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T07:30:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T07:30:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T07:30:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T07:30:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T07:30:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T07:30:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:17.001 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:30:17.158 DEBUG:teuthology.parallel:result is None 2021-11-20T07:30:17.223 DEBUG:teuthology.parallel:result is None 2021-11-20T07:30:17.224 INFO:teuthology.run_tasks:Running task install... 2021-11-20T07:30:17.231 DEBUG:teuthology.task.install:project ceph 2021-11-20T07:30:17.232 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'}} 2021-11-20T07:30:17.232 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T07:30:17.233 INFO:teuthology.task.install:Using flavor: default 2021-11-20T07:30:17.240 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-11-20T07:30:17.241 INFO:teuthology.task.install:extra packages: [] 2021-11-20T07:30:17.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-11-20T07:30:17.276 INFO:teuthology.orchestra.run.smithi071.stdout:CentOSStream 2021-11-20T07:30:17.278 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:30:17.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2021-11-20T07:30:17.280 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T07:30:17.281 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-11-20T07:30:17.314 INFO:teuthology.orchestra.run.smithi112.stdout:CentOSStream 2021-11-20T07:30:17.316 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:30:17.316 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T07:30:17.317 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2021-11-20T07:30:17.350 INFO:teuthology.orchestra.run.smithi071.stdout:CentOSStream 2021-11-20T07:30:17.352 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:30:17.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:30:17.353 INFO:teuthology.packaging:ref: None 2021-11-20T07:30:17.353 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:30:17.353 INFO:teuthology.packaging:branch: None 2021-11-20T07:30:17.354 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:30:17.567 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:30:17.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:30:17.622 INFO:teuthology.orchestra.run.smithi112.stdout:CentOSStream 2021-11-20T07:30:17.623 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:30:17.624 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:30:17.625 INFO:teuthology.packaging:ref: None 2021-11-20T07:30:17.625 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:30:17.625 INFO:teuthology.packaging:branch: None 2021-11-20T07:30:17.626 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:30:17.867 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:30:17.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:30:18.013 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T07:30:18.014 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T07:30:18.050 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T07:30:18.050 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:30:18.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T07:30:18.078 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T07:30:18.078 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:30:18.079 INFO:teuthology.packaging:ref: None 2021-11-20T07:30:18.079 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:30:18.079 INFO:teuthology.packaging:branch: None 2021-11-20T07:30:18.079 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:30:18.080 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T07:30:18.084 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T07:30:18.084 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T07:30:18.124 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T07:30:18.125 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:30:18.125 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T07:30:18.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T07:30:18.151 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T07:30:18.151 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:30:18.152 INFO:teuthology.packaging:ref: None 2021-11-20T07:30:18.152 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:30:18.152 INFO:teuthology.packaging:branch: None 2021-11-20T07:30:18.152 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:30:18.153 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T07:30:18.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T07:30:18.222 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T07:30:18.286 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2021-11-20T07:30:18.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2021-11-20T07:30:18.297 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T07:30:18.362 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2021-11-20T07:30:18.364 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2021-11-20T07:30:18.521 INFO:teuthology.orchestra.run.smithi071.stdout:75 files removed 2021-11-20T07:30:18.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2021-11-20T07:30:18.600 INFO:teuthology.orchestra.run.smithi112.stdout:75 files removed 2021-11-20T07:30:18.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2021-11-20T07:30:18.949 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2021-11-20T07:30:18.982 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 2.8 MB/s | 74 kB 00:00 2021-11-20T07:30:19.042 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 264 kB/s | 15 kB 00:00 2021-11-20T07:30:19.049 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 488 kB/s | 15 kB 00:00 2021-11-20T07:30:19.133 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 31 kB/s | 1.7 kB 00:00 2021-11-20T07:30:19.134 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 27 kB/s | 1.7 kB 00:00 2021-11-20T07:30:19.625 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - AppStream 37 MB/s | 17 MB 00:00 2021-11-20T07:30:19.630 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - AppStream 36 MB/s | 17 MB 00:00 2021-11-20T07:30:22.850 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - Base 45 MB/s | 15 MB 00:00 2021-11-20T07:30:22.872 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Base 50 MB/s | 15 MB 00:00 2021-11-20T07:30:25.062 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - Extras 222 kB/s | 16 kB 00:00 2021-11-20T07:30:25.066 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - Extras 213 kB/s | 16 kB 00:00 2021-11-20T07:30:25.286 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-8 - PowerTools 29 MB/s | 3.6 MB 00:00 2021-11-20T07:30:25.311 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-8 - PowerTools 25 MB/s | 3.6 MB 00:00 2021-11-20T07:30:28.258 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-11-20T07:30:28.272 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-11-20T07:30:29.403 INFO:teuthology.orchestra.run.smithi071.stdout:Apache Arrow 177 kB/s | 184 kB 00:01 2021-11-20T07:30:29.655 INFO:teuthology.orchestra.run.smithi112.stdout:Apache Arrow 144 kB/s | 184 kB 00:01 2021-11-20T07:30:30.758 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 11 MB 00:01 2021-11-20T07:30:31.002 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 11 MB 00:01 2021-11-20T07:30:33.198 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-11-20T07:30:33.398 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-11-20T07:30:35.942 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:30:35.945 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:35.945 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:30:35.945 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:35.946 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:30:35.946 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T07:30:35.946 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2021-11-20T07:30:35.946 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T07:30:35.947 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T07:30:35.947 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:30:35.947 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T07:30:35.947 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T07:30:35.948 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T07:30:35.948 INFO:teuthology.orchestra.run.smithi071.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T07:30:35.948 INFO:teuthology.orchestra.run.smithi071.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T07:30:35.948 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T07:30:35.949 INFO:teuthology.orchestra.run.smithi071.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T07:30:35.949 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T07:30:35.949 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T07:30:35.949 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T07:30:35.950 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T07:30:35.950 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T07:30:35.950 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T07:30:35.950 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T07:30:35.951 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T07:30:35.952 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T07:30:35.952 INFO:teuthology.orchestra.run.smithi071.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T07:30:35.952 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T07:30:35.953 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T07:30:35.953 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T07:30:35.953 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T07:30:35.953 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T07:30:35.954 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T07:30:35.954 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T07:30:35.954 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:35.954 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:30:35.955 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:30:35.955 INFO:teuthology.orchestra.run.smithi071.stdout:Install 25 Packages 2021-11-20T07:30:35.956 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2021-11-20T07:30:35.956 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:30:35.957 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 59 M 2021-11-20T07:30:35.957 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:30:36.166 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:30:36.168 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:36.169 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:30:36.169 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:36.169 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:30:36.169 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T07:30:36.170 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2021-11-20T07:30:36.170 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T07:30:36.170 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T07:30:36.170 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:30:36.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T07:30:36.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T07:30:36.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T07:30:36.171 INFO:teuthology.orchestra.run.smithi112.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T07:30:36.172 INFO:teuthology.orchestra.run.smithi112.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T07:30:36.172 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T07:30:36.172 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T07:30:36.172 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T07:30:36.172 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T07:30:36.173 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T07:30:36.173 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T07:30:36.173 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T07:30:36.174 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T07:30:36.174 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T07:30:36.174 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T07:30:36.174 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T07:30:36.175 INFO:teuthology.orchestra.run.smithi112.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T07:30:36.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T07:30:36.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T07:30:36.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T07:30:36.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T07:30:36.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T07:30:36.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T07:30:36.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T07:30:36.177 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:36.177 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:30:36.177 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:30:36.177 INFO:teuthology.orchestra.run.smithi112.stdout:Install 25 Packages 2021-11-20T07:30:36.177 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2021-11-20T07:30:36.178 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:30:36.178 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 59 M 2021-11-20T07:30:36.179 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:30:36.388 INFO:teuthology.orchestra.run.smithi071.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2021-11-20T07:30:36.422 INFO:teuthology.orchestra.run.smithi071.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 748 kB/s | 25 kB 00:00 2021-11-20T07:30:36.517 INFO:teuthology.orchestra.run.smithi071.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 801 kB 00:00 2021-11-20T07:30:36.547 INFO:teuthology.orchestra.run.smithi071.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 16 MB/s | 486 kB 00:00 2021-11-20T07:30:36.552 INFO:teuthology.orchestra.run.smithi112.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2021-11-20T07:30:36.564 INFO:teuthology.orchestra.run.smithi112.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 25 kB 00:00 2021-11-20T07:30:36.608 INFO:teuthology.orchestra.run.smithi112.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 18 MB/s | 801 kB 00:00 2021-11-20T07:30:36.640 INFO:teuthology.orchestra.run.smithi112.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 15 MB/s | 486 kB 00:00 2021-11-20T07:30:36.866 INFO:teuthology.orchestra.run.smithi112.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2021-11-20T07:30:36.998 INFO:teuthology.orchestra.run.smithi071.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 9.0 MB/s | 4.1 MB 00:00 2021-11-20T07:30:37.007 INFO:teuthology.orchestra.run.smithi071.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 4.8 MB/s | 44 kB 00:00 2021-11-20T07:30:37.018 INFO:teuthology.orchestra.run.smithi112.stdout:(6/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 15 MB/s | 13 MB 00:00 2021-11-20T07:30:37.019 INFO:teuthology.orchestra.run.smithi112.stdout:(7/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 292 kB/s | 44 kB 00:00 2021-11-20T07:30:37.045 INFO:teuthology.orchestra.run.smithi071.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.4 MB/s | 80 kB 00:00 2021-11-20T07:30:37.047 INFO:teuthology.orchestra.run.smithi112.stdout:(8/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.7 MB/s | 80 kB 00:00 2021-11-20T07:30:37.071 INFO:teuthology.orchestra.run.smithi112.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 213 kB 00:00 2021-11-20T07:30:37.087 INFO:teuthology.orchestra.run.smithi071.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 213 kB 00:00 2021-11-20T07:30:37.097 INFO:teuthology.orchestra.run.smithi112.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.7 MB/s | 387 kB 00:00 2021-11-20T07:30:37.126 INFO:teuthology.orchestra.run.smithi071.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.7 MB/s | 387 kB 00:00 2021-11-20T07:30:37.147 INFO:teuthology.orchestra.run.smithi071.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 18 MB/s | 368 kB 00:00 2021-11-20T07:30:37.154 INFO:teuthology.orchestra.run.smithi071.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 114 kB 00:00 2021-11-20T07:30:37.240 INFO:teuthology.orchestra.run.smithi112.stdout:(11/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2021-11-20T07:30:37.305 INFO:teuthology.orchestra.run.smithi112.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 548 kB/s | 114 kB 00:00 2021-11-20T07:30:37.307 INFO:teuthology.orchestra.run.smithi112.stdout:(13/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 368 kB 00:00 2021-11-20T07:30:37.338 INFO:teuthology.orchestra.run.smithi112.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 2021-11-20T07:30:37.341 INFO:teuthology.orchestra.run.smithi112.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 19 MB/s | 44 kB 00:00 2021-11-20T07:30:37.344 INFO:teuthology.orchestra.run.smithi112.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 8.9 MB/s | 354 kB 00:00 2021-11-20T07:30:37.344 INFO:teuthology.orchestra.run.smithi112.stdout:(17/27): daxctl-libs-71.1-2.el8.x86_64.rpm 13 MB/s | 41 kB 00:00 2021-11-20T07:30:37.347 INFO:teuthology.orchestra.run.smithi112.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 27 MB/s | 78 kB 00:00 2021-11-20T07:30:37.348 INFO:teuthology.orchestra.run.smithi112.stdout:(19/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 13 MB/s | 47 kB 00:00 2021-11-20T07:30:37.349 INFO:teuthology.orchestra.run.smithi112.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2021-11-20T07:30:37.357 INFO:teuthology.orchestra.run.smithi071.stdout:(12/27): ceph-common-16.2.4-0.el8.x86_64.rpm 17 MB/s | 24 MB 00:01 2021-11-20T07:30:37.366 INFO:teuthology.orchestra.run.smithi071.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 686 kB/s | 145 kB 00:00 2021-11-20T07:30:37.370 INFO:teuthology.orchestra.run.smithi071.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 61 MB/s | 259 kB 00:00 2021-11-20T07:30:37.372 INFO:teuthology.orchestra.run.smithi071.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2021-11-20T07:30:37.373 INFO:teuthology.orchestra.run.smithi071.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 34 MB/s | 41 kB 00:00 2021-11-20T07:30:37.375 INFO:teuthology.orchestra.run.smithi071.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 37 MB/s | 47 kB 00:00 2021-11-20T07:30:37.377 INFO:teuthology.orchestra.run.smithi071.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 48 MB/s | 78 kB 00:00 2021-11-20T07:30:37.475 INFO:teuthology.orchestra.run.smithi071.stdout:(19/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 13 MB 00:01 2021-11-20T07:30:37.482 INFO:teuthology.orchestra.run.smithi071.stdout:(20/27): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2021-11-20T07:30:37.487 INFO:teuthology.orchestra.run.smithi112.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-11-20T07:30:37.489 INFO:teuthology.orchestra.run.smithi112.stdout:(22/27): fmt-6.2.1-1.el8.x86_64.rpm 822 kB/s | 116 kB 00:00 2021-11-20T07:30:37.515 INFO:teuthology.orchestra.run.smithi112.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2021-11-20T07:30:37.519 INFO:teuthology.orchestra.run.smithi112.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2021-11-20T07:30:37.526 INFO:teuthology.orchestra.run.smithi112.stdout:(25/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2021-11-20T07:30:37.619 INFO:teuthology.orchestra.run.smithi071.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 482 kB/s | 116 kB 00:00 2021-11-20T07:30:37.629 INFO:teuthology.orchestra.run.smithi071.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2021-11-20T07:30:37.649 INFO:teuthology.orchestra.run.smithi071.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2021-11-20T07:30:37.649 INFO:teuthology.orchestra.run.smithi071.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 1.9 MB/s | 59 kB 00:00 2021-11-20T07:30:37.657 INFO:teuthology.orchestra.run.smithi071.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2021-11-20T07:30:37.704 INFO:teuthology.orchestra.run.smithi112.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2021-11-20T07:30:37.722 INFO:teuthology.orchestra.run.smithi112.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2021-11-20T07:30:37.726 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:30:37.726 INFO:teuthology.orchestra.run.smithi112.stdout:Total 38 MB/s | 59 MB 00:01 2021-11-20T07:30:37.737 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:30:37.759 INFO:teuthology.orchestra.run.smithi071.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 37 MB/s | 4.0 MB 00:00 2021-11-20T07:30:37.781 INFO:teuthology.orchestra.run.smithi071.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2021-11-20T07:30:37.784 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:30:37.785 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:30:37.786 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:30:37.786 INFO:teuthology.orchestra.run.smithi071.stdout:Total 32 MB/s | 59 MB 00:01 2021-11-20T07:30:37.799 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:30:37.845 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:30:37.845 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:30:38.115 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:30:38.119 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:30:38.176 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:30:38.179 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:30:39.266 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:30:39.302 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:30:39.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T07:30:39.475 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T07:30:39.587 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T07:30:39.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T07:30:39.736 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T07:30:39.810 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T07:30:40.145 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T07:30:40.219 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T07:30:40.294 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:30:40.368 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:30:40.393 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:30:40.498 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:30:40.610 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T07:30:40.685 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T07:30:40.777 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:30:40.836 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:30:40.846 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:30:40.901 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:30:41.008 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T07:30:41.050 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T07:30:41.144 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T07:30:41.201 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T07:30:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T07:30:41.632 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T07:30:41.669 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:30:41.736 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:30:41.776 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:30:41.857 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:30:41.931 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T07:30:42.036 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T07:30:42.052 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T07:30:42.143 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T07:30:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T07:30:42.214 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T07:30:42.274 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T07:30:42.333 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T07:30:42.436 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T07:30:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T07:30:42.676 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T07:30:42.819 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T07:30:42.939 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T07:30:43.076 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T07:30:43.183 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T07:30:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T07:30:43.326 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T07:30:43.485 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T07:30:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T07:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T07:30:43.805 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T07:30:43.917 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T07:30:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T07:30:44.013 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T07:30:44.119 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T07:30:44.185 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T07:30:44.196 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T07:30:44.386 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T07:30:44.393 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T07:30:44.526 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T07:30:44.581 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T07:30:44.610 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T07:30:44.792 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T07:30:44.812 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T07:30:44.951 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T07:30:45.019 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T07:30:45.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T07:30:45.393 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T07:30:49.280 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:49.485 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:50.185 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:50.490 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T07:30:50.586 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T07:30:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T07:30:50.742 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:30:50.913 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T07:30:51.007 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T07:30:51.132 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:31:08.605 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:31:08.606 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2021-11-20T07:31:08.606 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2021-11-20T07:31:08.606 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2021-11-20T07:31:08.606 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2021-11-20T07:31:08.607 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2021-11-20T07:31:08.607 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2021-11-20T07:31:08.607 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2021-11-20T07:31:08.607 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2021-11-20T07:31:08.608 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2021-11-20T07:31:08.608 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:08.827 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:31:08.827 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2021-11-20T07:31:08.827 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2021-11-20T07:31:08.827 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2021-11-20T07:31:08.828 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2021-11-20T07:31:08.828 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2021-11-20T07:31:08.828 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2021-11-20T07:31:08.828 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2021-11-20T07:31:08.829 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2021-11-20T07:31:08.829 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2021-11-20T07:31:08.829 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:09.904 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T07:31:09.976 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T07:31:09.977 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T07:31:10.117 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T07:31:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T07:31:10.177 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T07:31:10.201 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T07:31:10.201 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:10.364 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T07:31:10.364 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:10.926 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:10.927 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T07:31:10.927 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T07:31:10.927 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T07:31:10.928 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:31:10.928 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T07:31:10.929 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:31:10.929 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T07:31:10.929 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T07:31:10.929 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:31:10.930 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T07:31:10.930 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T07:31:10.930 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T07:31:10.930 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T07:31:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T07:31:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T07:31:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T07:31:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T07:31:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T07:31:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T07:31:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T07:31:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T07:31:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T07:31:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T07:31:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T07:31:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T07:31:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:31:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T07:31:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T07:31:11.121 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:11.121 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T07:31:11.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T07:31:11.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T07:31:11.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T07:31:11.122 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T07:31:11.123 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T07:31:11.123 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T07:31:11.123 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T07:31:11.123 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T07:31:11.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T07:31:11.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T07:31:11.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T07:31:11.124 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T07:31:11.125 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T07:31:11.125 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T07:31:11.125 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T07:31:11.125 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T07:31:11.126 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T07:31:11.126 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T07:31:11.128 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T07:31:11.128 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T07:31:11.128 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T07:31:11.129 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T07:31:11.129 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T07:31:11.129 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T07:31:11.130 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T07:31:11.131 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T07:31:11.131 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T07:31:11.819 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:11.819 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:11.819 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2021-11-20T07:31:11.820 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.820 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:11.820 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T07:31:11.822 INFO:teuthology.orchestra.run.smithi112.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T07:31:11.822 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T07:31:11.822 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T07:31:11.822 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.823 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T07:31:11.823 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T07:31:11.823 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T07:31:11.823 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.823 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T07:31:11.824 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.824 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T07:31:11.824 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T07:31:11.824 INFO:teuthology.orchestra.run.smithi112.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T07:31:11.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T07:31:11.825 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.826 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.826 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.826 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:11.826 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:11.969 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T07:31:11.969 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:11.970 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2021-11-20T07:31:11.970 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.970 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:11.970 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.971 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.972 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.972 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.972 INFO:teuthology.orchestra.run.smithi071.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T07:31:11.972 INFO:teuthology.orchestra.run.smithi071.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T07:31:11.972 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T07:31:11.973 INFO:teuthology.orchestra.run.smithi071.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T07:31:11.973 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.973 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T07:31:11.973 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T07:31:11.974 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T07:31:11.974 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.974 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T07:31:11.974 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.975 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T07:31:11.975 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T07:31:11.975 INFO:teuthology.orchestra.run.smithi071.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T07:31:11.975 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.975 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.976 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.976 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T07:31:11.976 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.976 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.977 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:11.977 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:11.977 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:12.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2021-11-20T07:31:12.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2021-11-20T07:31:12.634 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:39 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:12.796 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:39 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:13.898 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:13.899 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:13.899 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:31:13.900 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:13.900 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:31:13.900 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T07:31:13.900 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:31:13.900 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T07:31:13.901 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T07:31:13.901 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T07:31:13.901 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T07:31:13.901 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T07:31:13.901 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:13.902 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:31:13.902 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:13.902 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2021-11-20T07:31:13.902 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:13.903 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 53 M 2021-11-20T07:31:13.903 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 229 M 2021-11-20T07:31:13.904 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:31:13.948 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 4.3 MB/s | 202 kB 00:00 2021-11-20T07:31:13.951 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 54 MB/s | 187 kB 00:00 2021-11-20T07:31:13.956 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 65 MB/s | 323 kB 00:00 2021-11-20T07:31:14.004 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 169 kB 00:00 2021-11-20T07:31:14.072 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 597 kB/s | 69 kB 00:00 2021-11-20T07:31:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:14.082 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:31:14.083 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:14.083 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:31:14.084 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T07:31:14.084 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:31:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T07:31:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T07:31:14.085 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T07:31:14.086 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T07:31:14.086 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T07:31:14.086 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:14.087 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:31:14.087 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:14.088 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2021-11-20T07:31:14.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:14.089 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 53 M 2021-11-20T07:31:14.089 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 229 M 2021-11-20T07:31:14.089 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:31:14.122 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 5.8 MB/s | 169 kB 00:00 2021-11-20T07:31:14.134 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 3.8 MB/s | 202 kB 00:00 2021-11-20T07:31:14.145 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 31 MB/s | 323 kB 00:00 2021-11-20T07:31:14.149 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.9 MB/s | 187 kB 00:00 2021-11-20T07:31:14.261 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 597 kB/s | 69 kB 00:00 2021-11-20T07:31:15.116 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 43 MB/s | 52 MB 00:01 2021-11-20T07:31:15.120 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:15.120 INFO:teuthology.orchestra.run.smithi112.stdout:Total 43 MB/s | 53 MB 00:01 2021-11-20T07:31:15.126 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:31:15.151 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:31:15.151 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:31:15.394 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 40 MB/s | 52 MB 00:01 2021-11-20T07:31:15.398 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:15.399 INFO:teuthology.orchestra.run.smithi071.stdout:Total 40 MB/s | 53 MB 00:01 2021-11-20T07:31:15.409 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:31:15.430 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:31:15.430 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:31:15.434 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:31:15.437 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:31:15.713 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:31:15.717 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:31:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:31:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T07:31:16.269 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:31:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T07:31:16.431 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T07:31:16.547 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T07:31:16.624 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T07:31:16.631 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T07:31:16.802 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T07:31:16.805 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T07:31:16.883 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T07:31:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T07:31:17.051 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T07:31:17.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T07:31:21.745 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T07:31:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T07:31:22.005 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T07:31:22.225 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T07:31:22.261 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T07:31:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T07:31:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T07:31:22.262 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T07:31:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T07:31:22.263 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T07:31:22.481 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T07:31:22.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T07:31:22.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T07:31:22.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T07:31:22.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T07:31:22.482 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T07:31:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T07:31:22.648 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:22.649 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T07:31:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T07:31:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T07:31:22.649 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:22.650 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:22.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2021-11-20T07:31:22.984 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T07:31:22.985 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:22.985 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:22.985 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T07:31:22.985 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T07:31:22.986 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T07:31:22.986 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:22.986 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:23.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2021-11-20T07:31:23.376 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:50 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:23.701 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:50 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:24.648 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:24.652 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:24.652 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:31:24.653 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:24.653 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:31:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T07:31:24.653 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:31:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T07:31:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T07:31:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T07:31:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T07:31:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T07:31:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T07:31:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T07:31:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T07:31:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T07:31:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T07:31:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T07:31:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T07:31:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T07:31:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T07:31:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T07:31:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T07:31:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T07:31:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T07:31:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T07:31:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T07:31:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T07:31:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T07:31:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T07:31:24.660 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T07:31:24.660 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T07:31:24.660 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T07:31:24.660 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T07:31:24.661 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T07:31:24.661 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T07:31:24.661 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T07:31:24.661 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T07:31:24.662 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:24.662 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:31:24.663 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:24.663 INFO:teuthology.orchestra.run.smithi112.stdout:Install 32 Packages 2021-11-20T07:31:24.663 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:24.664 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 31 M 2021-11-20T07:31:24.664 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 111 M 2021-11-20T07:31:24.665 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:31:24.708 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 124 kB/s | 6.3 kB 00:00 2021-11-20T07:31:24.830 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2021-11-20T07:31:24.928 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 8.8 MB/s | 2.4 MB 00:00 2021-11-20T07:31:25.011 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:25.013 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 2.5 MB/s | 220 kB 00:00 2021-11-20T07:31:25.014 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:25.015 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:31:25.015 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:25.015 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:31:25.015 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T07:31:25.016 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:31:25.017 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T07:31:25.017 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T07:31:25.017 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T07:31:25.017 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T07:31:25.018 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T07:31:25.018 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T07:31:25.018 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T07:31:25.019 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T07:31:25.019 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T07:31:25.019 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T07:31:25.019 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T07:31:25.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T07:31:25.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T07:31:25.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T07:31:25.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T07:31:25.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T07:31:25.021 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T07:31:25.021 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T07:31:25.021 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T07:31:25.021 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T07:31:25.022 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T07:31:25.022 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T07:31:25.022 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T07:31:25.023 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T07:31:25.023 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T07:31:25.024 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T07:31:25.024 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T07:31:25.025 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T07:31:25.025 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T07:31:25.025 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T07:31:25.025 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T07:31:25.026 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:25.026 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:31:25.026 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:25.027 INFO:teuthology.orchestra.run.smithi071.stdout:Install 32 Packages 2021-11-20T07:31:25.027 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:25.028 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 M 2021-11-20T07:31:25.028 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 111 M 2021-11-20T07:31:25.029 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:31:25.041 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 287 kB/s | 6.3 kB 00:00 2021-11-20T07:31:25.053 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2021-11-20T07:31:25.063 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.2 MB/s | 157 kB 00:00 2021-11-20T07:31:25.064 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 44 MB/s | 103 kB 00:00 2021-11-20T07:31:25.066 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 35 MB/s | 54 kB 00:00 2021-11-20T07:31:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 75 MB/s | 457 kB 00:00 2021-11-20T07:31:25.076 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 68 MB/s | 237 kB 00:00 2021-11-20T07:31:25.084 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 72 MB/s | 559 kB 00:00 2021-11-20T07:31:25.085 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-markupsafe-0.23-19.el8.x86_64. 1.2 MB/s | 39 kB 00:00 2021-11-20T07:31:25.086 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 47 MB/s | 109 kB 00:00 2021-11-20T07:31:25.102 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2021-11-20T07:31:25.129 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2021-11-20T07:31:25.145 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 14 MB/s | 220 kB 00:00 2021-11-20T07:31:25.176 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 5.0 MB/s | 157 kB 00:00 2021-11-20T07:31:25.178 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 21 MB/s | 39 kB 00:00 2021-11-20T07:31:25.180 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 43 MB/s | 103 kB 00:00 2021-11-20T07:31:25.182 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 33 MB/s | 54 kB 00:00 2021-11-20T07:31:25.189 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 73 MB/s | 457 kB 00:00 2021-11-20T07:31:25.192 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 65 MB/s | 237 kB 00:00 2021-11-20T07:31:25.198 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 384 kB/s | 44 kB 00:00 2021-11-20T07:31:25.201 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 67 MB/s | 559 kB 00:00 2021-11-20T07:31:25.204 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 37 MB/s | 109 kB 00:00 2021-11-20T07:31:25.229 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2021-11-20T07:31:25.237 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2021-11-20T07:31:25.284 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2021-11-20T07:31:25.288 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 6.3 MB/s | 384 kB 00:00 2021-11-20T07:31:25.313 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 389 kB/s | 11 kB 00:00 2021-11-20T07:31:25.317 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 617 kB/s | 18 kB 00:00 2021-11-20T07:31:25.319 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 380 kB/s | 44 kB 00:00 2021-11-20T07:31:25.342 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2021-11-20T07:31:25.347 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T07:31:25.376 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 535 kB/s | 16 kB 00:00 2021-11-20T07:31:25.400 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2021-11-20T07:31:25.405 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 651 kB/s | 19 kB 00:00 2021-11-20T07:31:25.477 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 49 MB/s | 18 MB 00:00 2021-11-20T07:31:25.478 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 768 kB/s | 185 kB 00:00 2021-11-20T07:31:25.479 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2021-11-20T07:31:25.506 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 400 kB/s | 11 kB 00:00 2021-11-20T07:31:25.507 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 631 kB/s | 18 kB 00:00 2021-11-20T07:31:25.535 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-20T07:31:25.535 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T07:31:25.536 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 26 MB/s | 18 MB 00:00 2021-11-20T07:31:25.537 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 175 kB/s | 24 kB 00:00 2021-11-20T07:31:25.537 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 222 kB/s | 29 kB 00:00 2021-11-20T07:31:25.564 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 555 kB/s | 16 kB 00:00 2021-11-20T07:31:25.570 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.5 MB/s | 251 kB 00:00 2021-11-20T07:31:25.571 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 6.9 MB/s | 241 kB 00:00 2021-11-20T07:31:25.591 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-11-20T07:31:25.592 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 674 kB/s | 19 kB 00:00 2021-11-20T07:31:25.600 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.8 MB/s | 85 kB 00:00 2021-11-20T07:31:25.601 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 776 kB/s | 23 kB 00:00 2021-11-20T07:31:25.619 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 858 kB/s | 24 kB 00:00 2021-11-20T07:31:25.620 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T07:31:25.621 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): python3-trustme-0.6.0-4.el8.noarch.rpm 326 kB/s | 27 kB 00:00 2021-11-20T07:31:25.625 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:25.626 INFO:teuthology.orchestra.run.smithi112.stdout:Total 32 MB/s | 31 MB 00:00 2021-11-20T07:31:25.639 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.3 MB/s | 384 kB 00:00 2021-11-20T07:31:25.640 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:31:25.648 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 981 kB/s | 27 kB 00:00 2021-11-20T07:31:25.670 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 8.3 MB/s | 251 kB 00:00 2021-11-20T07:31:25.677 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-11-20T07:31:25.677 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.8 MB/s | 85 kB 00:00 2021-11-20T07:31:25.680 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:31:25.681 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:31:25.699 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 814 kB/s | 23 kB 00:00 2021-11-20T07:31:25.701 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:25.702 INFO:teuthology.orchestra.run.smithi071.stdout:Total 45 MB/s | 31 MB 00:00 2021-11-20T07:31:25.716 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:31:25.753 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:31:25.754 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:31:25.897 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:31:25.901 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:31:25.971 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:31:25.974 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:31:27.090 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:31:27.136 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:31:27.297 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T07:31:27.366 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T07:31:27.521 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T07:31:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T07:31:27.788 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T07:31:27.807 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T07:31:28.018 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T07:31:28.023 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T07:31:28.143 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T07:31:28.204 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T07:31:28.349 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T07:31:28.387 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T07:31:28.558 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T07:31:28.602 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T07:31:28.720 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T07:31:28.739 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T07:31:28.951 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T07:31:28.975 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T07:31:29.196 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T07:31:29.231 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T07:31:29.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T07:31:29.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T07:31:29.692 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T07:31:29.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T07:31:30.142 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T07:31:30.150 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T07:31:30.392 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T07:31:30.395 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T07:31:30.607 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T07:31:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T07:31:30.869 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T07:31:30.879 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T07:31:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T07:31:31.126 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T07:31:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T07:31:31.462 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T07:31:31.639 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T07:31:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T07:31:31.818 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T07:31:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T07:31:32.029 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T07:31:32.111 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T07:31:32.318 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T07:31:32.369 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T07:31:32.579 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T07:31:32.661 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T07:31:32.837 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T07:31:32.910 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T07:31:33.192 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T07:31:33.247 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T07:31:33.522 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T07:31:33.623 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T07:31:33.866 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T07:31:33.997 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T07:31:34.070 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T07:31:34.201 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T07:31:35.588 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T07:31:35.723 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T07:31:35.811 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T07:31:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T07:31:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T07:31:36.416 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T07:31:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T07:31:36.642 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T07:31:36.699 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T07:31:36.864 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T07:31:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T07:31:36.898 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T07:31:37.047 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T07:31:37.070 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T07:31:37.087 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T07:31:37.202 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T07:31:37.445 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T07:31:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T07:31:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T07:31:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T07:31:37.446 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T07:31:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T07:31:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T07:31:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T07:31:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T07:31:37.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T07:31:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T07:31:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T07:31:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T07:31:37.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T07:31:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T07:31:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T07:31:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T07:31:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T07:31:37.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T07:31:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T07:31:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T07:31:37.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T07:31:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T07:31:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T07:31:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T07:31:37.451 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T07:31:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T07:31:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T07:31:37.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T07:31:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T07:31:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T07:31:37.453 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T07:31:37.619 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T07:31:37.619 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T07:31:37.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T07:31:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T07:31:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T07:31:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T07:31:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T07:31:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T07:31:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T07:31:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T07:31:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T07:31:37.623 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T07:31:37.623 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T07:31:37.623 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T07:31:37.623 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T07:31:37.624 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T07:31:37.624 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T07:31:37.624 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T07:31:37.624 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T07:31:37.624 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T07:31:37.625 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T07:31:37.625 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T07:31:37.625 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T07:31:37.625 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T07:31:37.626 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T07:31:37.626 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T07:31:37.626 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T07:31:37.626 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T07:31:37.626 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T07:31:37.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T07:31:37.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T07:31:37.627 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T07:31:38.189 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T07:31:38.190 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:38.190 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:38.190 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.191 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.191 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.191 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T07:31:38.191 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.192 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.192 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T07:31:38.192 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T07:31:38.192 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T07:31:38.193 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T07:31:38.193 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T07:31:38.193 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T07:31:38.193 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T07:31:38.194 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T07:31:38.194 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T07:31:38.194 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T07:31:38.194 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T07:31:38.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T07:31:38.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T07:31:38.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T07:31:38.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T07:31:38.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T07:31:38.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T07:31:38.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T07:31:38.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T07:31:38.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T07:31:38.197 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T07:31:38.197 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T07:31:38.197 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T07:31:38.197 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T07:31:38.198 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T07:31:38.198 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T07:31:38.198 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:38.199 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:38.420 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T07:31:38.421 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:38.421 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:38.422 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.422 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.422 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.422 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T07:31:38.423 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.423 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:38.423 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T07:31:38.423 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T07:31:38.424 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T07:31:38.424 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T07:31:38.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T07:31:38.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T07:31:38.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T07:31:38.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T07:31:38.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T07:31:38.431 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T07:31:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T07:31:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T07:31:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T07:31:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T07:31:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T07:31:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T07:31:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T07:31:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T07:31:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T07:31:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T07:31:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T07:31:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T07:31:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T07:31:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T07:31:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T07:31:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T07:31:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:38.436 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:38.438 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2021-11-20T07:31:38.645 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2021-11-20T07:31:38.977 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:39.185 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:39.984 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:31:40.162 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:31:40.271 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:40.272 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:31:40.272 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:40.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2021-11-20T07:31:40.449 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:40.450 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:31:40.450 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:40.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2021-11-20T07:31:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:07 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:41.056 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:08 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:42.188 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:42.189 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:42.189 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:31:42.190 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:42.190 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:31:42.190 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T07:31:42.190 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:31:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:31:42.191 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:42.191 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 71 k 2021-11-20T07:31:42.192 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 290 k 2021-11-20T07:31:42.192 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:31:42.219 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-16.2.4-0.el8.noarch.rpm 2.4 MB/s | 71 kB 00:00 2021-11-20T07:31:42.220 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:42.221 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.3 MB/s | 71 kB 00:00 2021-11-20T07:31:42.221 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:31:42.228 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:31:42.228 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:31:42.230 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:31:42.233 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:31:42.284 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:31:42.332 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:42.334 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:31:42.335 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:42.335 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:31:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T07:31:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:42.336 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:31:42.336 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:42.336 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:31:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:42.337 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 71 k 2021-11-20T07:31:42.337 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 290 k 2021-11-20T07:31:42.337 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:31:42.346 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.9 MB/s | 71 kB 00:00 2021-11-20T07:31:42.347 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:42.347 INFO:teuthology.orchestra.run.smithi112.stdout:Total 5.3 MB/s | 71 kB 00:00 2021-11-20T07:31:42.348 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:31:42.355 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:31:42.355 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:31:42.356 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:31:42.360 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:31:42.413 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:31:45.054 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.339 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.528 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.533 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.725 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.857 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:45.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:45.858 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:45.859 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T07:31:45.859 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:45.859 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:46.040 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T07:31:46.041 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:46.041 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:46.041 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T07:31:46.041 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:46.042 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:46.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T07:31:46.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T07:31:46.609 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:46.810 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:47.916 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:47.917 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:47.918 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2021-11-20T07:31:47.918 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:47.918 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:31:47.918 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T07:31:47.919 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:31:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:31:47.920 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:47.920 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 163 k 2021-11-20T07:31:47.920 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 587 k 2021-11-20T07:31:47.920 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:31:47.964 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.5 MB/s | 163 kB 00:00 2021-11-20T07:31:47.965 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:47.965 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.4 MB/s | 163 kB 00:00 2021-11-20T07:31:47.966 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:31:47.979 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:31:47.979 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:31:48.028 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:31:48.031 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:31:48.141 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:31:48.162 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:48.164 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:48.164 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2021-11-20T07:31:48.164 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:48.164 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:31:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T07:31:48.165 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:48.165 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:31:48.165 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:48.165 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:31:48.166 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:48.166 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 163 k 2021-11-20T07:31:48.166 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 587 k 2021-11-20T07:31:48.166 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:31:48.177 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T07:31:48.178 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:48.178 INFO:teuthology.orchestra.run.smithi112.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T07:31:48.179 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:31:48.192 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:31:48.192 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:31:48.240 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:31:48.244 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:31:48.354 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:48.362 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:31:48.577 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:48.731 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:48.981 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:49.300 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:49.300 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:49.301 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:49.302 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:50.324 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:31:50.324 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:50.325 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:50.325 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T07:31:50.326 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:50.326 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:51.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2021-11-20T07:31:51.577 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:51.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2021-11-20T07:31:52.174 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:19 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:52.623 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:31:52.912 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:52.913 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:31:52.913 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:52.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T07:31:53.177 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:31:53.462 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:53.463 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:31:53.464 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:53.518 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:20 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:53.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T07:31:54.088 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:21 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:54.854 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:31:54.855 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:54.855 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:31:54.856 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:54.856 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:31:54.856 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T07:31:54.856 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:31:54.857 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T07:31:54.857 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T07:31:54.857 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T07:31:54.857 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T07:31:54.857 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T07:31:54.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:54.858 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:31:54.858 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:31:54.858 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2021-11-20T07:31:54.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:54.859 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.0 M 2021-11-20T07:31:54.859 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 15 M 2021-11-20T07:31:54.859 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:31:54.881 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.0 MB/s | 24 kB 00:00 2021-11-20T07:31:54.931 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 843 kB/s | 43 kB 00:00 2021-11-20T07:31:54.958 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 89 kB/s | 8.9 kB 00:00 2021-11-20T07:31:54.969 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 24 MB/s | 2.7 MB 00:00 2021-11-20T07:31:55.013 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 410 kB/s | 34 kB 00:00 2021-11-20T07:31:55.105 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-11-20T07:31:55.107 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:55.108 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 3.0 MB 00:00 2021-11-20T07:31:55.111 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:31:55.119 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:31:55.120 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:31:55.191 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:31:55.194 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:31:55.375 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:31:55.376 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:31:55.376 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:55.377 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:31:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T07:31:55.377 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:31:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T07:31:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T07:31:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T07:31:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T07:31:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T07:31:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:55.379 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:31:55.379 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:31:55.379 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2021-11-20T07:31:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:55.381 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.0 M 2021-11-20T07:31:55.381 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 15 M 2021-11-20T07:31:55.381 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:31:55.399 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.1 MB/s | 24 kB 00:00 2021-11-20T07:31:55.407 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 322 kB/s | 8.9 kB 00:00 2021-11-20T07:31:55.430 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 52 MB/s | 2.7 MB 00:00 2021-11-20T07:31:55.434 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2021-11-20T07:31:55.456 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:31:55.490 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 403 kB/s | 34 kB 00:00 2021-11-20T07:31:55.574 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-11-20T07:31:55.577 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:31:55.578 INFO:teuthology.orchestra.run.smithi112.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-20T07:31:55.580 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:31:55.589 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:31:55.589 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:31:55.662 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:31:55.665 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:31:55.735 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T07:31:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:31:55.971 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T07:31:56.154 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T07:31:56.242 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T07:31:56.259 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T07:31:56.520 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T07:31:56.745 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T07:31:56.900 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T07:31:57.345 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T07:31:57.648 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:31:57.885 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:31:57.885 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T07:31:57.885 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T07:31:57.886 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T07:31:57.886 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T07:31:57.887 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T07:31:57.969 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T07:31:58.211 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T07:31:58.212 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:58.212 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:31:58.212 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.212 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.212 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.213 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T07:31:58.213 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T07:31:58.213 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T07:31:58.213 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:31:58.213 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:31:58.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T07:31:58.297 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:31:58.550 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:31:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T07:31:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T07:31:58.551 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T07:31:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T07:31:58.552 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T07:31:58.829 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:25 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:31:58.856 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T07:31:58.857 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:58.857 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:31:58.857 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.857 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.857 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T07:31:58.858 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T07:31:58.858 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T07:31:58.858 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T07:31:58.858 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:31:58.859 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:31:58.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T07:31:59.479 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:26 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:00.173 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:00.174 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:32:00.174 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:32:00.174 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:32:00.174 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T07:32:00.175 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:32:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T07:32:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T07:32:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T07:32:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T07:32:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T07:32:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T07:32:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T07:32:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:00.177 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:00.177 INFO:teuthology.orchestra.run.smithi071.stdout:===================================================================================================== 2021-11-20T07:32:00.177 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2021-11-20T07:32:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:00.178 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2021-11-20T07:32:00.179 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 211 M 2021-11-20T07:32:00.179 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:00.293 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 41 MB/s | 4.7 MB 00:00 2021-11-20T07:32:00.393 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): ceph-mgr-diskprediction-local-16.2.4-0.e 40 MB/s | 8.6 MB 00:00 2021-11-20T07:32:00.413 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2021-11-20T07:32:00.427 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2021-11-20T07:32:00.450 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 28 MB/s | 643 kB 00:00 2021-11-20T07:32:00.458 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 3.4 MB/s | 225 kB 00:00 2021-11-20T07:32:00.461 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): libquadmath-8.5.0-3.el8.x86_64.rpm 16 MB/s | 170 kB 00:00 2021-11-20T07:32:00.666 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 54 MB/s | 14 MB 00:00 2021-11-20T07:32:00.670 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:00.671 INFO:teuthology.orchestra.run.smithi071.stdout:Total 74 MB/s | 37 MB 00:00 2021-11-20T07:32:00.685 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:00.705 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:00.705 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:00.776 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:00.777 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:32:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:32:00.777 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:32:00.778 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T07:32:00.778 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:32:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T07:32:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T07:32:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T07:32:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T07:32:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T07:32:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T07:32:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T07:32:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:00.781 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:00.781 INFO:teuthology.orchestra.run.smithi112.stdout:===================================================================================================== 2021-11-20T07:32:00.781 INFO:teuthology.orchestra.run.smithi112.stdout:Install 8 Packages 2021-11-20T07:32:00.781 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:00.782 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 37 M 2021-11-20T07:32:00.782 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 211 M 2021-11-20T07:32:00.783 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:00.884 INFO:teuthology.orchestra.run.smithi112.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 45 MB/s | 4.7 MB 00:00 2021-11-20T07:32:00.910 INFO:teuthology.orchestra.run.smithi112.stdout:(2/8): ceph-mgr-diskprediction-local-16.2.4-0.e 66 MB/s | 8.6 MB 00:00 2021-11-20T07:32:00.966 INFO:teuthology.orchestra.run.smithi112.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2021-11-20T07:32:00.989 INFO:teuthology.orchestra.run.smithi112.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 2.8 MB/s | 225 kB 00:00 2021-11-20T07:32:01.010 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:01.014 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:01.031 INFO:teuthology.orchestra.run.smithi112.stdout:(5/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2021-11-20T07:32:01.043 INFO:teuthology.orchestra.run.smithi112.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 14 MB/s | 170 kB 00:00 2021-11-20T07:32:01.088 INFO:teuthology.orchestra.run.smithi112.stdout:(7/8): libgfortran-8.5.0-3.el8.x86_64.rpm 6.4 MB/s | 643 kB 00:00 2021-11-20T07:32:01.220 INFO:teuthology.orchestra.run.smithi112.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 54 MB/s | 14 MB 00:00 2021-11-20T07:32:01.225 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:01.225 INFO:teuthology.orchestra.run.smithi112.stdout:Total 82 MB/s | 37 MB 00:00 2021-11-20T07:32:01.239 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:01.258 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:01.258 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:01.553 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:01.562 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:01.566 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:01.726 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T07:32:01.866 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T07:32:02.042 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T07:32:02.403 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:02.644 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T07:32:02.727 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T07:32:02.845 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T07:32:03.025 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T07:32:03.059 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:03.622 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T07:32:03.663 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:03.825 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T07:32:03.833 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:04.447 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:04.527 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T07:32:04.658 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T07:32:04.885 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T07:32:05.158 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T07:32:05.378 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T07:32:05.776 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T07:32:06.066 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T07:32:06.971 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T07:32:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T07:32:08.295 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T07:32:08.882 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T07:32:09.187 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T07:32:09.187 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T07:32:09.187 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T07:32:09.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:09.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T07:32:09.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T07:32:09.188 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T07:32:09.189 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T07:32:09.564 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T07:32:09.577 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T07:32:09.578 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:09.578 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:09.578 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T07:32:09.578 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T07:32:09.579 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T07:32:09.579 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T07:32:09.579 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T07:32:09.579 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T07:32:09.580 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T07:32:09.580 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T07:32:09.580 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:09.580 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:09.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2021-11-20T07:32:09.748 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T07:32:10.064 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T07:32:10.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T07:32:10.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T07:32:10.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T07:32:10.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T07:32:10.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T07:32:10.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T07:32:10.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T07:32:10.213 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:10.441 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T07:32:10.441 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:10.442 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:10.442 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T07:32:10.442 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T07:32:10.442 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T07:32:10.443 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T07:32:10.443 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T07:32:10.443 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T07:32:10.443 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T07:32:10.443 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T07:32:10.444 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:10.444 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:10.536 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2021-11-20T07:32:11.075 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:38 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:11.516 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:11.518 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:11.518 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:32:11.518 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:11.519 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:11.519 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T07:32:11.519 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:32:11.519 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T07:32:11.519 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T07:32:11.520 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T07:32:11.520 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T07:32:11.520 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T07:32:11.520 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T07:32:11.521 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T07:32:11.521 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T07:32:11.521 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T07:32:11.521 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T07:32:11.521 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T07:32:11.522 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T07:32:11.522 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:11.522 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:11.522 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:11.523 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2021-11-20T07:32:11.523 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:11.524 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2021-11-20T07:32:11.525 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 24 M 2021-11-20T07:32:11.525 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:11.543 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-11-20T07:32:11.556 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 770 kB/s | 27 kB 00:00 2021-11-20T07:32:11.558 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 47 MB/s | 110 kB 00:00 2021-11-20T07:32:11.561 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 57 MB/s | 155 kB 00:00 2021-11-20T07:32:11.562 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-jsonpointer-1.10-11.el8.noarch. 490 kB/s | 20 kB 00:00 2021-11-20T07:32:11.563 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 34 MB/s | 43 kB 00:00 2021-11-20T07:32:11.578 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.5 MB/s | 126 kB 00:00 2021-11-20T07:32:11.644 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 201 kB/s | 16 kB 00:00 2021-11-20T07:32:11.649 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 379 kB/s | 33 kB 00:00 2021-11-20T07:32:11.693 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 706 kB/s | 81 kB 00:00 2021-11-20T07:32:11.707 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2021-11-20T07:32:11.723 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.0 MB/s | 61 kB 00:00 2021-11-20T07:32:11.891 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.7 MB/s | 1.2 MB 00:00 2021-11-20T07:32:11.894 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:11.895 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.1 MB/s | 1.9 MB 00:00 2021-11-20T07:32:11.904 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:11.918 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:11.918 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:11.997 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:12.000 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:12.422 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:12.423 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:12.424 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:32:12.424 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:12.424 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:12.424 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T07:32:12.425 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:32:12.425 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T07:32:12.425 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T07:32:12.425 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T07:32:12.426 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T07:32:12.426 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T07:32:12.426 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T07:32:12.426 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T07:32:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T07:32:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T07:32:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T07:32:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T07:32:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T07:32:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:12.428 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:12.428 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:12.429 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2021-11-20T07:32:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:12.430 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.9 M 2021-11-20T07:32:12.430 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2021-11-20T07:32:12.430 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:12.447 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T07:32:12.463 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 764 kB/s | 27 kB 00:00 2021-11-20T07:32:12.464 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 53 MB/s | 110 kB 00:00 2021-11-20T07:32:12.468 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 498 kB/s | 20 kB 00:00 2021-11-20T07:32:12.468 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 51 MB/s | 155 kB 00:00 2021-11-20T07:32:12.469 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 31 MB/s | 43 kB 00:00 2021-11-20T07:32:12.480 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.6 MB/s | 126 kB 00:00 2021-11-20T07:32:12.523 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:12.550 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 202 kB/s | 16 kB 00:00 2021-11-20T07:32:12.552 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 392 kB/s | 33 kB 00:00 2021-11-20T07:32:12.593 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 724 kB/s | 81 kB 00:00 2021-11-20T07:32:12.610 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T07:32:12.622 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.0 MB/s | 61 kB 00:00 2021-11-20T07:32:12.728 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T07:32:12.797 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.7 MB/s | 1.2 MB 00:00 2021-11-20T07:32:12.800 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:12.801 INFO:teuthology.orchestra.run.smithi112.stdout:Total 5.1 MB/s | 1.9 MB 00:00 2021-11-20T07:32:12.810 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:12.824 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:12.825 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:12.901 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:12.904 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:12.938 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T07:32:13.177 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T07:32:13.380 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T07:32:13.403 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:13.622 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T07:32:13.654 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T07:32:13.843 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T07:32:13.899 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T07:32:14.110 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T07:32:14.214 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T07:32:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T07:32:14.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T07:32:14.615 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T07:32:14.893 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T07:32:15.189 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T07:32:15.458 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T07:32:15.512 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T07:32:15.796 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T07:32:15.963 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T07:32:16.172 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T07:32:16.306 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T07:32:16.581 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T07:32:16.581 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T07:32:16.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T07:32:16.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T07:32:16.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T07:32:16.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T07:32:16.582 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T07:32:16.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T07:32:16.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T07:32:16.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T07:32:16.583 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T07:32:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T07:32:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T07:32:16.586 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T07:32:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T07:32:17.011 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T07:32:17.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:17.011 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:17.012 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T07:32:17.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T07:32:17.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T07:32:17.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T07:32:17.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T07:32:17.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T07:32:17.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T07:32:17.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T07:32:17.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T07:32:17.014 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T07:32:17.014 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T07:32:17.014 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T07:32:17.015 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T07:32:17.015 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:17.016 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:17.080 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T07:32:17.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T07:32:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T07:32:17.459 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T07:32:17.642 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:44 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:17.731 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T07:32:17.732 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T07:32:17.732 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T07:32:17.732 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T07:32:17.732 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T07:32:17.733 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T07:32:17.733 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T07:32:17.733 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T07:32:17.733 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T07:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T07:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T07:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T07:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T07:32:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T07:32:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:18.171 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T07:32:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T07:32:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T07:32:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T07:32:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T07:32:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T07:32:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T07:32:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T07:32:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T07:32:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T07:32:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T07:32:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T07:32:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T07:32:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:18.176 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:18.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T07:32:18.818 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:18.965 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:18.966 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:18.966 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2021-11-20T07:32:18.966 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:18.967 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:18.967 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T07:32:18.967 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:32:18.967 INFO:teuthology.orchestra.run.smithi071.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T07:32:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T07:32:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T07:32:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T07:32:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T07:32:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:18.969 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:18.969 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:18.969 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2021-11-20T07:32:18.969 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:18.970 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 5.8 M 2021-11-20T07:32:18.970 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 24 M 2021-11-20T07:32:18.971 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:19.000 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.8 MB/s | 89 kB 00:00 2021-11-20T07:32:19.017 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 11 MB/s | 538 kB 00:00 2021-11-20T07:32:19.066 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-11-20T07:32:19.084 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 252 kB/s | 21 kB 00:00 2021-11-20T07:32:19.217 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 324 kB/s | 49 kB 00:00 2021-11-20T07:32:19.272 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2021-11-20T07:32:19.275 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:19.275 INFO:teuthology.orchestra.run.smithi071.stdout:Total 19 MB/s | 5.8 MB 00:00 2021-11-20T07:32:19.335 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:19.345 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:19.345 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:19.431 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:19.434 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:19.760 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:20.036 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T07:32:20.105 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:20.106 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:20.106 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2021-11-20T07:32:20.106 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:20.107 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:20.107 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T07:32:20.107 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:32:20.107 INFO:teuthology.orchestra.run.smithi112.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T07:32:20.108 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T07:32:20.108 INFO:teuthology.orchestra.run.smithi112.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T07:32:20.108 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T07:32:20.108 INFO:teuthology.orchestra.run.smithi112.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T07:32:20.108 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:20.109 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:20.109 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:20.109 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2021-11-20T07:32:20.109 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:20.110 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 5.8 M 2021-11-20T07:32:20.110 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2021-11-20T07:32:20.111 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:20.129 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.5 MB/s | 89 kB 00:00 2021-11-20T07:32:20.175 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.9 MB/s | 538 kB 00:00 2021-11-20T07:32:20.204 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2021-11-20T07:32:20.212 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 251 kB/s | 21 kB 00:00 2021-11-20T07:32:20.279 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T07:32:20.317 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 430 kB/s | 49 kB 00:00 2021-11-20T07:32:20.345 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2021-11-20T07:32:20.347 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:20.348 INFO:teuthology.orchestra.run.smithi112.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-11-20T07:32:20.352 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:20.363 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:20.363 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:20.447 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:20.450 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:21.061 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T07:32:21.195 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T07:32:21.337 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T07:32:21.576 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T07:32:21.803 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T07:32:22.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:32:22.281 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:32:22.281 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T07:32:22.282 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T07:32:22.282 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T07:32:22.282 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T07:32:22.282 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T07:32:22.307 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T07:32:22.604 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T07:32:22.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:22.605 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:22.605 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T07:32:22.605 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T07:32:22.605 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T07:32:22.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:22.606 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:22.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2021-11-20T07:32:22.711 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T07:32:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T07:32:23.205 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:32:23.229 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:50 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:23.464 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T07:32:23.464 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T07:32:23.464 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T07:32:23.464 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T07:32:23.465 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T07:32:23.465 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T07:32:23.789 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T07:32:23.789 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:23.789 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:23.789 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T07:32:23.790 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T07:32:23.790 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T07:32:23.790 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:23.791 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:23.895 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2021-11-20T07:32:24.433 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:24.571 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:24.572 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:24.573 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:24.573 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:24.573 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:24.574 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T07:32:24.574 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2021-11-20T07:32:24.574 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T07:32:24.574 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:24.575 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:24.575 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:24.575 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2021-11-20T07:32:24.575 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:24.576 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 967 k 2021-11-20T07:32:24.576 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.2 M 2021-11-20T07:32:24.577 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:24.610 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T07:32:24.634 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2021-11-20T07:32:24.636 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:24.636 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 967 kB 00:00 2021-11-20T07:32:24.638 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:24.660 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:24.660 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:24.741 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:24.744 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:24.912 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:25.213 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T07:32:25.428 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T07:32:25.753 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:25.754 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:25.755 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:25.755 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T07:32:25.755 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2021-11-20T07:32:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T07:32:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:25.756 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:25.756 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:25.757 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2021-11-20T07:32:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:25.757 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 967 k 2021-11-20T07:32:25.757 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 3.2 M 2021-11-20T07:32:25.757 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:25.779 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 36 MB/s | 885 kB 00:00 2021-11-20T07:32:25.791 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T07:32:25.791 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T07:32:25.793 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T07:32:25.794 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:25.794 INFO:teuthology.orchestra.run.smithi112.stdout:Total 24 MB/s | 967 kB 00:00 2021-11-20T07:32:25.796 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:25.818 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:25.818 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:25.899 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:25.902 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:26.073 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:26.150 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T07:32:26.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:26.151 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:26.151 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T07:32:26.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:26.151 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:26.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2021-11-20T07:32:26.397 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T07:32:26.646 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T07:32:26.858 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:26.994 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T07:32:26.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T07:32:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T07:32:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:27.345 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T07:32:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:27.346 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:27.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2021-11-20T07:32:28.084 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:28.176 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:28.177 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:28.177 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:28.178 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:28.178 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:28.178 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T07:32:28.179 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:28.179 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:28.179 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:28.180 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:32:28.180 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:28.180 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 131 k 2021-11-20T07:32:28.180 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 439 k 2021-11-20T07:32:28.181 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:28.199 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 6.0 MB/s | 131 kB 00:00 2021-11-20T07:32:28.200 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:28.200 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.6 MB/s | 131 kB 00:00 2021-11-20T07:32:28.201 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:28.208 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:28.208 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:28.259 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:28.262 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:28.394 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:28.644 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:28.909 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:29.308 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:29.308 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:29.313 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:29.313 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:29.314 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:29.314 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:29.400 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:29.401 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:29.401 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:29.402 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:29.402 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:29.402 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T07:32:29.402 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:29.403 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:29.403 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:29.403 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:32:29.403 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:29.403 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 131 k 2021-11-20T07:32:29.404 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 439 k 2021-11-20T07:32:29.404 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:29.429 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 4.7 MB/s | 131 kB 00:00 2021-11-20T07:32:29.430 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:29.431 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.4 MB/s | 131 kB 00:00 2021-11-20T07:32:29.431 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:29.438 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:29.489 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:29.492 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:29.496 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2021-11-20T07:32:29.708 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:29.920 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:30.043 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:57 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:30.210 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:30.619 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:30.620 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:30.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2021-11-20T07:32:31.097 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:31.340 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:31.391 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:31.393 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:31.393 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:31.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2021-11-20T07:32:32.006 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:32.388 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:32.682 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:32.683 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:32.683 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:32.756 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2021-11-20T07:32:33.296 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:33.328 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:33.329 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:33.330 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:33.330 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T07:32:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:33.331 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:33.331 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:33.331 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:32:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:33.332 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 24 k 2021-11-20T07:32:33.332 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 94 k 2021-11-20T07:32:33.332 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:33.345 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 24 kB 00:00 2021-11-20T07:32:33.345 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:33.346 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.5 MB/s | 24 kB 00:00 2021-11-20T07:32:33.346 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:33.350 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:33.350 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:33.352 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:33.355 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:33.609 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:33.841 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:34.164 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:34.164 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:34.165 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:34.165 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:34.165 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:34.165 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:34.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2021-11-20T07:32:34.620 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:34.622 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:34.622 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:34.622 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:34.622 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:34.623 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T07:32:34.623 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:34.623 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:34.623 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:34.624 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:32:34.624 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:34.624 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 24 k 2021-11-20T07:32:34.624 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 94 k 2021-11-20T07:32:34.624 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:34.653 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 1.1 MB/s | 24 kB 00:00 2021-11-20T07:32:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:Total 806 kB/s | 24 kB 00:00 2021-11-20T07:32:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:34.657 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:34.657 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:34.659 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:34.662 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:34.786 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:01 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:35.479 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:35.912 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:36.160 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:36.453 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:36.454 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:36.455 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:36.455 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:36.455 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:36.456 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:36.541 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2021-11-20T07:32:36.792 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:37.077 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:37.085 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:37.086 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:37.086 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:37.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2021-11-20T07:32:38.054 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:38.178 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:38.344 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:38.499 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:38.499 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:38.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2021-11-20T07:32:39.043 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:06 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:39.212 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:39.504 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:39.504 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:39.504 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:39.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2021-11-20T07:32:40.071 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:40.113 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:40.361 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:40.362 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:40.362 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:40.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2021-11-20T07:32:40.977 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:41.095 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:41.389 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:41.390 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:41.390 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:41.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2021-11-20T07:32:41.999 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:08 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:42.579 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:42.866 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:42.867 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:42.868 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:42.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2021-11-20T07:32:42.987 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:43.275 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:43.276 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:43.276 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:43.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2021-11-20T07:32:43.486 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:43.872 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:44.503 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:44.791 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:44.793 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:44.793 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:44.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2021-11-20T07:32:44.894 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:45.187 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:45.188 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:45.189 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:45.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2021-11-20T07:32:45.412 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:45.787 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:46.411 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:46.703 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:46.704 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:46.705 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:46.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2021-11-20T07:32:46.819 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:47.107 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:47.108 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2021-11-20T07:32:47.108 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:47.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2021-11-20T07:32:47.329 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:47.712 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:48.311 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T07:32:48.600 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:48.601 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2021-11-20T07:32:48.601 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:48.733 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2021-11-20T07:32:49.157 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:49.159 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:49.160 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:49.160 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T07:32:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:49.161 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:49.161 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:49.161 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:32:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:49.162 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 89 k 2021-11-20T07:32:49.162 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 298 k 2021-11-20T07:32:49.162 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:49.172 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 6.4 MB/s | 89 kB 00:00 2021-11-20T07:32:49.173 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:49.173 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.8 MB/s | 89 kB 00:00 2021-11-20T07:32:49.174 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:49.185 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:49.185 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:49.235 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:49.240 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:49.269 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:16 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:49.384 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:49.549 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:50.140 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:50.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:50.141 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:50.141 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:50.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:50.141 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:50.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2021-11-20T07:32:50.549 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:50.550 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:50.551 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:50.551 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T07:32:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:50.552 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:50.552 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:50.552 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:32:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:50.553 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 89 k 2021-11-20T07:32:50.553 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 298 k 2021-11-20T07:32:50.553 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:50.562 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.3 MB/s | 89 kB 00:00 2021-11-20T07:32:50.563 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:50.563 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.6 MB/s | 89 kB 00:00 2021-11-20T07:32:50.564 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:50.575 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:50.576 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:50.627 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:50.630 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:50.735 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:50.869 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:17 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:50.952 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:51.202 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:51.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:51.511 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:51.511 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:51.512 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:51.512 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:51.512 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:51.710 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2021-11-20T07:32:52.152 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:52.153 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:52.154 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:52.154 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:52.154 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:52.154 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T07:32:52.155 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:52.155 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:52.155 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:52.155 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:32:52.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:52.156 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.7 M 2021-11-20T07:32:52.156 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 18 M 2021-11-20T07:32:52.156 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:52.255 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:19 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:52.256 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2021-11-20T07:32:52.256 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:52.257 INFO:teuthology.orchestra.run.smithi071.stdout:Total 36 MB/s | 3.7 MB 00:00 2021-11-20T07:32:52.258 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:52.271 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:52.271 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:52.335 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:52.339 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:53.023 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:53.700 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:53.701 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:53.701 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:53.701 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:53.701 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:53.702 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:53.771 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:53.772 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:53.772 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:53.772 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:53.773 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:53.773 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T07:32:53.773 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:53.773 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:53.773 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:53.774 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:32:53.774 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:53.774 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.7 M 2021-11-20T07:32:53.774 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 18 M 2021-11-20T07:32:53.775 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:53.832 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 62 MB/s | 3.7 MB 00:00 2021-11-20T07:32:53.833 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:53.834 INFO:teuthology.orchestra.run.smithi112.stdout:Total 60 MB/s | 3.7 MB 00:00 2021-11-20T07:32:53.834 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:53.848 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:53.848 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:53.912 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:53.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2021-11-20T07:32:53.916 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:54.382 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:54.460 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:21 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:55.022 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:55.347 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:55.347 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:55.348 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:55.348 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:55.569 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2021-11-20T07:32:55.756 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2021-11-20T07:32:55.758 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:55.758 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:55.759 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:55.759 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2021-11-20T07:32:55.760 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T07:32:55.760 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2021-11-20T07:32:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2021-11-20T07:32:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2021-11-20T07:32:55.762 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:55.762 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 173 k 2021-11-20T07:32:55.764 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 649 k 2021-11-20T07:32:55.764 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2021-11-20T07:32:55.804 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2021-11-20T07:32:55.805 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:55.805 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.6 MB/s | 173 kB 00:00 2021-11-20T07:32:55.805 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2021-11-20T07:32:55.817 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2021-11-20T07:32:55.818 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2021-11-20T07:32:55.868 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2021-11-20T07:32:55.871 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2021-11-20T07:32:55.982 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2021-11-20T07:32:56.117 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:23 ago on Sat 20 Nov 2021 07:30:33 AM UTC. 2021-11-20T07:32:56.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:56.448 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:56.814 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:56.814 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2021-11-20T07:32:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:56.815 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:32:56.815 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2021-11-20T07:32:56.964 DEBUG:teuthology.parallel:result is None 2021-11-20T07:32:57.391 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2021-11-20T07:32:57.392 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2021-11-20T07:32:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2021-11-20T07:32:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T07:32:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:57.394 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2021-11-20T07:32:57.394 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2021-11-20T07:32:57.394 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2021-11-20T07:32:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:57.395 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 173 k 2021-11-20T07:32:57.395 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 649 k 2021-11-20T07:32:57.395 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2021-11-20T07:32:57.406 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 12 MB/s | 173 kB 00:00 2021-11-20T07:32:57.408 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2021-11-20T07:32:57.408 INFO:teuthology.orchestra.run.smithi112.stdout:Total 11 MB/s | 173 kB 00:00 2021-11-20T07:32:57.408 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2021-11-20T07:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2021-11-20T07:32:57.421 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2021-11-20T07:32:57.471 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2021-11-20T07:32:57.474 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2021-11-20T07:32:57.591 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2021-11-20T07:32:57.816 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:58.077 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T07:32:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:58.416 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2021-11-20T07:32:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T07:32:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:32:58.417 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2021-11-20T07:32:58.716 DEBUG:teuthology.parallel:result is None 2021-11-20T07:32:58.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo lsb_release -is 2021-11-20T07:32:58.759 INFO:teuthology.orchestra.run.smithi071.stdout:CentOSStream 2021-11-20T07:32:58.760 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:32:58.761 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:32:58.761 INFO:teuthology.packaging:ref: None 2021-11-20T07:32:58.762 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:32:58.762 INFO:teuthology.packaging:branch: None 2021-11-20T07:32:58.762 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:32:58.964 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:32:58.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:32:59.176 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T07:32:59.202 INFO:teuthology.orchestra.run.smithi071.stdout:16.2.4-0.el8 2021-11-20T07:32:59.202 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T07:32:59.203 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T07:32:59.204 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2021-11-20T07:32:59.239 INFO:teuthology.orchestra.run.smithi112.stdout:CentOSStream 2021-11-20T07:32:59.241 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T07:32:59.241 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T07:32:59.241 INFO:teuthology.packaging:ref: None 2021-11-20T07:32:59.242 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T07:32:59.242 INFO:teuthology.packaging:branch: None 2021-11-20T07:32:59.243 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T07:32:59.451 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:32:59.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T07:32:59.656 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T07:32:59.679 INFO:teuthology.orchestra.run.smithi112.stdout:16.2.4-0.el8 2021-11-20T07:32:59.679 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T07:32:59.680 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T07:32:59.680 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-11-20T07:32:59.681 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:32:59.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T07:32:59.711 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:32:59.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T07:32:59.745 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-11-20T07:32:59.745 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:32:59.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T07:32:59.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T07:32:59.846 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:32:59.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T07:32:59.872 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T07:32:59.936 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-11-20T07:32:59.937 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:32:59.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T07:32:59.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T07:33:00.031 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:33:00.032 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T07:33:00.056 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T07:33:00.121 INFO:teuthology.run_tasks:Running task cephadm... 2021-11-20T07:33:00.212 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T07:33:00.213 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2021-11-20T07:33:00.214 INFO:tasks.cephadm:Cluster fsid is 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:00.214 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-11-20T07:33:00.215 INFO:tasks.cephadm:No mon roles; fabricating mons 2021-11-20T07:33:00.215 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi071': '172.21.15.71', 'mon.smithi112': '172.21.15.112'} 2021-11-20T07:33:00.215 INFO:tasks.cephadm:Normalizing hostnames... 2021-11-20T07:33:00.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname $(hostname -s) 2021-11-20T07:33:00.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2021-11-20T07:33:00.268 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2021-11-20T07:33:00.269 DEBUG:teuthology.orchestra.run.smithi071:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.525 INFO:teuthology.orchestra.run.smithi071.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 07:33 /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.526 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.636 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 07:33 /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.637 DEBUG:teuthology.orchestra.run.smithi071:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.657 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T07:33:00.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2021-11-20T07:33:00.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2021-11-20T07:33:00.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /etc/ceph 2021-11-20T07:33:00.808 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2021-11-20T07:33:00.833 INFO:tasks.cephadm:Writing seed config... 2021-11-20T07:33:00.835 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2021-11-20T07:33:00.835 INFO:tasks.cephadm: override: [client] debug client = 20 2021-11-20T07:33:00.835 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-11-20T07:33:00.836 INFO:tasks.cephadm: override: [client] rados mon op timeout = 15m 2021-11-20T07:33:00.836 INFO:tasks.cephadm: override: [client] rados osd op timeout = 15m 2021-11-20T07:33:00.836 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2021-11-20T07:33:00.836 INFO:tasks.cephadm: override: [mds] debug mds = 20 2021-11-20T07:33:00.837 INFO:tasks.cephadm: override: [mds] debug ms = 1 2021-11-20T07:33:00.837 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2021-11-20T07:33:00.837 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2021-11-20T07:33:00.838 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2021-11-20T07:33:00.838 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2021-11-20T07:33:00.838 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2021-11-20T07:33:00.839 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 15m 2021-11-20T07:33:00.839 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 15m 2021-11-20T07:33:00.839 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-11-20T07:33:00.840 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-11-20T07:33:00.840 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-11-20T07:33:00.840 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-11-20T07:33:00.840 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-11-20T07:33:00.841 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2021-11-20T07:33:00.841 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2021-11-20T07:33:00.841 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2021-11-20T07:33:00.842 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2021-11-20T07:33:00.842 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2021-11-20T07:33:00.842 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2021-11-20T07:33:00.843 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2021-11-20T07:33:00.843 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2021-11-20T07:33:00.843 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-11-20T07:33:00.843 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-11-20T07:33:00.844 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2021-11-20T07:33:00.844 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2021-11-20T07:33:00.844 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2021-11-20T07:33:00.845 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2021-11-20T07:33:00.845 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2021-11-20T07:33:00.845 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2021-11-20T07:33:00.845 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2021-11-20T07:33:00.846 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:33:00.847 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-11-20T07:33:00.865 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 16910ea2-49d4-11ec-8c2c-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 15m rados osd op timeout = 15m [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 15m rados osd op timeout = 15m 2021-11-20T07:33:00.867 DEBUG:teuthology.orchestra.run.smithi071:mon.smithi071> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service 2021-11-20T07:33:00.910 INFO:tasks.cephadm:Bootstrapping... 2021-11-20T07:33:00.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.71 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:33:00.938 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:-- Logs begin at Sat 2021-11-20 07:22:38 UTC. -- 2021-11-20T07:33:01.169 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T07:33:02.077 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: 3.4.1-dev 2021-11-20T07:33:02.148 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying podman|docker is present... 2021-11-20T07:33:02.148 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T07:33:02.190 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: 3.4.1-dev 2021-11-20T07:33:02.291 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying lvm2 is present... 2021-11-20T07:33:02.292 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying time synchronization is in place... 2021-11-20T07:33:02.292 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T07:33:02.299 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T07:33:02.299 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active chrony.service 2021-11-20T07:33:02.305 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:33:02.306 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T07:33:02.312 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: enabled 2021-11-20T07:33:02.313 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T07:33:02.318 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: active 2021-11-20T07:33:02.319 INFO:teuthology.orchestra.run.smithi071.stderr:Unit chronyd.service is enabled and running 2021-11-20T07:33:02.319 INFO:teuthology.orchestra.run.smithi071.stderr:Repeating the final host check... 2021-11-20T07:33:02.319 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T07:33:02.358 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: 3.4.1-dev 2021-11-20T07:33:02.462 INFO:teuthology.orchestra.run.smithi071.stderr:podman|docker (/bin/podman) is present 2021-11-20T07:33:02.462 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl is present 2021-11-20T07:33:02.463 INFO:teuthology.orchestra.run.smithi071.stderr:lvcreate is present 2021-11-20T07:33:02.463 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T07:33:02.468 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T07:33:02.468 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active chrony.service 2021-11-20T07:33:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:33:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T07:33:02.480 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: enabled 2021-11-20T07:33:02.480 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T07:33:02.485 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: active 2021-11-20T07:33:02.487 INFO:teuthology.orchestra.run.smithi071.stderr:Unit chronyd.service is enabled and running 2021-11-20T07:33:02.487 INFO:teuthology.orchestra.run.smithi071.stderr:Host looks OK 2021-11-20T07:33:02.487 INFO:teuthology.orchestra.run.smithi071.stderr:Cluster fsid: 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:02.487 INFO:teuthology.orchestra.run.smithi071.stderr:Acquiring lock 139769234262504 on /run/cephadm/16910ea2-49d4-11ec-8c2c-001a4aab830c.lock 2021-11-20T07:33:02.488 INFO:teuthology.orchestra.run.smithi071.stderr:Lock 139769234262504 acquired on /run/cephadm/16910ea2-49d4-11ec-8c2c-001a4aab830c.lock 2021-11-20T07:33:02.488 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying IP 172.21.15.71 port 3300 ... 2021-11-20T07:33:02.489 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying IP 172.21.15.71 port 6789 ... 2021-11-20T07:33:02.489 INFO:teuthology.orchestra.run.smithi071.stderr:Base mon IP is 172.21.15.71, final addrv is [v2:172.21.15.71:3300,v1:172.21.15.71:6789] 2021-11-20T07:33:02.489 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /sbin/ip route ls 2021-11-20T07:33:02.490 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2021-11-20T07:33:02.490 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.71 metric 100 2021-11-20T07:33:02.491 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /sbin/ip -6 route ls 2021-11-20T07:33:02.493 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-11-20T07:33:02.493 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-11-20T07:33:02.493 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2021-11-20T07:33:02.494 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /sbin/ip -6 addr ls 2021-11-20T07:33:02.495 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-11-20T07:33:02.495 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-11-20T07:33:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T07:33:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-11-20T07:33:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d765/64 scope link 2021-11-20T07:33:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T07:33:02.497 INFO:teuthology.orchestra.run.smithi071.stderr:Mon IP 172.21.15.71 is in CIDR network 172.21.0.0/20 2021-11-20T07:33:02.497 INFO:teuthology.orchestra.run.smithi071.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-11-20T07:33:02.498 INFO:teuthology.orchestra.run.smithi071.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2021-11-20T07:33:02.498 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2021-11-20T07:33:02.567 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2021-11-20T07:33:02.925 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Getting image source signatures 2021-11-20T07:33:02.932 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T07:33:02.932 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T07:33:03.056 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T07:33:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T07:33:15.170 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T07:33:15.254 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Writing manifest to image destination 2021-11-20T07:33:15.254 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: Storing signatures 2021-11-20T07:33:15.551 INFO:teuthology.orchestra.run.smithi071.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T07:33:15.572 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2021-11-20T07:33:16.128 INFO:teuthology.orchestra.run.smithi071.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T07:33:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T07:33:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:Extracting ceph user uid/gid from container image... 2021-11-20T07:33:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2021-11-20T07:33:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:stat: 167 167 2021-11-20T07:33:17.255 INFO:teuthology.orchestra.run.smithi071.stderr:Creating initial keys... 2021-11-20T07:33:17.256 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T07:33:17.682 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-authtool: AQC9pJhhIyXgJxAACFBpyEjhLeZTCyvv3FudNQ== 2021-11-20T07:33:17.998 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T07:33:18.365 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-authtool: AQC+pJhhBGQ/FRAA8bZ1jXznIShRP/HS5ZQYYA== 2021-11-20T07:33:18.637 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T07:33:19.015 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-authtool: AQC/pJhh/7MdABAA4EpZXx3TDlPv+ptPG/cJfw== 2021-11-20T07:33:19.276 INFO:teuthology.orchestra.run.smithi071.stderr:Creating initial monmap... 2021-11-20T07:33:19.277 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpycfw5cb1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --addv smithi071 [v2:172.21.15.71:3300,v1:172.21.15.71:6789] /tmp/monmap 2021-11-20T07:33:19.665 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T07:33:19.665 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:19.666 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T07:33:20.023 INFO:teuthology.orchestra.run.smithi071.stderr:monmaptool for smithi071 [v2:172.21.15.71:3300,v1:172.21.15.71:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T07:33:20.023 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/monmaptool: set fsid to 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:20.023 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T07:33:20.024 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:33:20.024 INFO:teuthology.orchestra.run.smithi071.stderr:Creating mon... 2021-11-20T07:33:20.024 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071:/var/lib/ceph/mon/ceph-smithi071:z -v /tmp/ceph-tmpnn9wlz2i:/tmp/keyring:z -v /tmp/ceph-tmpycfw5cb1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi071 --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-11-20T07:33:20.409 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.393+0000 7f62f654d700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T07:33:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.393+0000 7f62f654d700 1 imported monmap: 2021-11-20T07:33:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: epoch 0 2021-11-20T07:33:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: last_changed 2021-11-20T07:33:19.655860+0000 2021-11-20T07:33:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: created 2021-11-20T07:33:19.655860+0000 2021-11-20T07:33:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-11-20T07:33:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-11-20T07:33:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.smithi071 2021-11-20T07:33:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.393+0000 7f62f654d700 0 /usr/bin/ceph-mon: set fsid to 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:20.413 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T07:33:20.413 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.413 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T07:33:20.413 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Compile date May 13 2021 2021-11-20T07:33:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: DB SUMMARY 2021-11-20T07:33:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi071/store.db dir, Total Num: 0, files: 2021-11-20T07:33:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi071/store.db: 2021-11-20T07:33:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T07:33:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.create_if_missing: 1 2021-11-20T07:33:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T07:33:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.env: 0x55ba26f7b1c0 2021-11-20T07:33:20.416 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.fs: Posix File System 2021-11-20T07:33:20.416 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.info_log: 0x55ba288acec0 2021-11-20T07:33:20.416 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T07:33:20.416 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.statistics: (nil) 2021-11-20T07:33:20.417 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.use_fsync: 0 2021-11-20T07:33:20.417 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T07:33:20.417 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T07:33:20.417 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T07:33:20.445 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T07:33:20.445 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T07:33:20.445 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T07:33:20.446 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T07:33:20.446 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T07:33:20.446 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T07:33:20.446 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T07:33:20.447 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T07:33:20.447 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.db_log_dir: 2021-11-20T07:33:20.447 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi071/store.db 2021-11-20T07:33:20.447 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T07:33:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T07:33:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T07:33:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T07:33:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T07:33:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T07:33:20.449 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T07:33:20.449 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T07:33:20.449 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T07:33:20.449 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T07:33:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.write_buffer_manager: 0x55ba288b3530 2021-11-20T07:33:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T07:33:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T07:33:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T07:33:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T07:33:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T07:33:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T07:33:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T07:33:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T07:33:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T07:33:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.unordered_write: 0 2021-11-20T07:33:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T07:33:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T07:33:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T07:33:20.453 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T07:33:20.453 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.row_cache: None 2021-11-20T07:33:20.453 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.wal_filter: None 2021-11-20T07:33:20.453 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T07:33:20.454 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T07:33:20.454 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T07:33:20.454 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T07:33:20.454 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T07:33:20.454 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T07:33:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T07:33:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T07:33:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T07:33:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T07:33:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T07:33:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T07:33:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T07:33:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T07:33:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T07:33:20.457 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T07:33:20.457 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T07:33:20.457 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T07:33:20.457 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T07:33:20.458 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T07:33:20.458 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T07:33:20.458 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.max_open_files: -1 2021-11-20T07:33:20.458 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T07:33:20.458 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T07:33:20.459 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T07:33:20.459 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T07:33:20.459 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Compression algorithms supported: 2021-11-20T07:33:20.459 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T07:33:20.460 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kZSTD supported: 0 2021-11-20T07:33:20.460 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T07:33:20.460 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T07:33:20.460 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T07:33:20.461 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T07:33:20.461 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T07:33:20.461 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T07:33:20.461 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.394+0000 7f62f654d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T07:33:20.461 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.395+0000 7f62f654d700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-11-20T07:33:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi071/store.db/MANIFEST-000001 2021-11-20T07:33:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T07:33:20.462 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.463 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T07:33:20.463 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.merge_operator: 2021-11-20T07:33:20.463 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_filter: None 2021-11-20T07:33:20.463 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T07:33:20.464 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T07:33:20.464 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T07:33:20.464 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ba287c4120) 2021-11-20T07:33:20.464 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-11-20T07:33:20.464 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T07:33:20.465 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T07:33:20.465 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-11-20T07:33:20.465 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: index_type: 0 2021-11-20T07:33:20.465 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-11-20T07:33:20.465 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-11-20T07:33:20.466 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-11-20T07:33:20.466 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-11-20T07:33:20.466 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: checksum: 1 2021-11-20T07:33:20.466 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-11-20T07:33:20.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_cache: 0x55ba287fad10 2021-11-20T07:33:20.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-11-20T07:33:20.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-11-20T07:33:20.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-11-20T07:33:20.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-11-20T07:33:20.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-11-20T07:33:20.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-11-20T07:33:20.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-11-20T07:33:20.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-11-20T07:33:20.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-11-20T07:33:20.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-11-20T07:33:20.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-11-20T07:33:20.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-11-20T07:33:20.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-11-20T07:33:20.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-11-20T07:33:20.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-11-20T07:33:20.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T07:33:20.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-11-20T07:33:20.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-11-20T07:33:20.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-11-20T07:33:20.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: format_version: 2 2021-11-20T07:33:20.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-11-20T07:33:20.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: block_align: 0 2021-11-20T07:33:20.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T07:33:20.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T07:33:20.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression: NoCompression 2021-11-20T07:33:20.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T07:33:20.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T07:33:20.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T07:33:20.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.num_levels: 7 2021-11-20T07:33:20.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T07:33:20.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T07:33:20.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T07:33:20.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T07:33:20.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T07:33:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T07:33:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T07:33:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:33:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T07:33:20.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T07:33:20.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T07:33:20.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T07:33:20.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T07:33:20.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:33:20.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T07:33:20.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T07:33:20.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T07:33:20.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T07:33:20.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T07:33:20.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T07:33:20.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T07:33:20.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T07:33:20.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T07:33:20.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T07:33:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T07:33:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T07:33:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T07:33:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T07:33:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T07:33:20.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T07:33:20.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T07:33:20.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T07:33:20.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T07:33:20.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T07:33:20.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T07:33:20.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T07:33:20.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T07:33:20.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T07:33:20.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T07:33:20.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T07:33:20.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T07:33:20.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T07:33:20.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T07:33:20.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T07:33:20.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T07:33:20.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T07:33:20.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T07:33:20.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T07:33:20.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T07:33:20.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T07:33:20.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T07:33:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T07:33:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T07:33:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T07:33:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T07:33:20.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T07:33:20.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T07:33:20.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T07:33:20.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T07:33:20.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.ttl: 2592000 2021-11-20T07:33:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.400+0000 7f62f654d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T07:33:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.401+0000 7f62f654d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi071/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T07:33:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.401+0000 7f62f654d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T07:33:20.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.402+0000 7f62f654d700 4 rocksdb: DB pointer 0x55ba288bf800 2021-11-20T07:33:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.402+0000 7f62df3e6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T07:33:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.402+0000 7f62df3e6700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T07:33:20.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-11-20T07:33:20.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:20.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T07:33:20.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T07:33:20.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:33:20.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T07:33:20.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T07:33:20.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:33:20.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T07:33:20.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:20.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:20.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:33:20.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:33:20.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T07:33:20.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:20.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:20.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:20.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:20.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:20.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:33:20.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:33:20.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:33:20.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:20.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:20.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:33:20.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:33:20.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T07:33:20.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:20.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:20.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:33:20.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:33:20.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T07:33:20.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:20.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:20.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:20.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:20.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:20.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:33:20.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:33:20.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:33:20.499 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:20.499 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:20.499 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:33:20.499 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.499 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:33:20.500 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: 2021-11-20T07:33:20.500 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.404+0000 7f62f654d700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-11-20T07:33:20.500 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.404+0000 7f62f654d700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-11-20T07:33:20.500 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph-mon: debug 2021-11-20T07:33:20.404+0000 7f62f654d700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi071 for mon.smithi071 2021-11-20T07:33:20.663 INFO:teuthology.orchestra.run.smithi071.stderr:create mon.smithi071 on 2021-11-20T07:33:20.664 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:20.670 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl disable ceph.target 2021-11-20T07:33:20.676 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-11-20T07:33:20.759 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl enable ceph.target 2021-11-20T07:33:20.766 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-11-20T07:33:20.846 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl start ceph.target 2021-11-20T07:33:20.852 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl enable ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target 2021-11-20T07:33:20.858 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target. 2021-11-20T07:33:20.859 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target. 2021-11-20T07:33:20.940 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl start ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target 2021-11-20T07:33:20.947 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl daemon-reload 2021-11-20T07:33:21.032 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:33:21.052 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl reset-failed ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:33:21.057 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Failed to reset failed state of unit ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service: Unit ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service not loaded. 2021-11-20T07:33:21.058 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl enable ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:33:21.064 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Created symlink /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target.wants/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service → /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@.service. 2021-11-20T07:33:21.143 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl start ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:33:21.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 systemd[1]: Starting Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:33:21.668 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 podman[32953]: 2021-11-20T07:33:21.668 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 bash[32874]: dfdc92d1c764e6cc6f89d3aa0900e6e9ce54146e807d63cff675d20e0816088d 2021-11-20T07:33:21.668 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.630713+0000 mon.smithi071 (mon.0) 0 : [INF] mkfs 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:21.669 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.604278+0000 mon.smithi071 (mon.0) 1 : cluster [INF] mon.smithi071 is new leader, mons smithi071 in quorum (ranks 0) 2021-11-20T07:33:21.670 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T07:33:21.679 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: disabled 2021-11-20T07:33:21.680 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T07:33:21.686 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:33:21.687 INFO:teuthology.orchestra.run.smithi071.stderr:firewalld.service is not enabled 2021-11-20T07:33:21.687 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T07:33:21.687 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T07:33:21.688 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mon to start... 2021-11-20T07:33:21.688 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mon... 2021-11-20T07:33:21.688 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071:/var/lib/ceph/mon/ceph-smithi071:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2021-11-20T07:33:21.927 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:21 smithi071 systemd[1]: Started Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c. 2021-11-20T07:33:22.115 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.109+0000 7fb0f45bb700 1 Processor -- start 2021-11-20T07:33:22.116 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.109+0000 7fb0f45bb700 1 -- start start 2021-11-20T07:33:22.116 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.109+0000 7fb0f45bb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:22.116 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.110+0000 7fb0f45bb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0fb480 con 0x7fb0ec0faae0 2021-11-20T07:33:22.116 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.110+0000 7fb0f2357700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:22.117 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.110+0000 7fb0f2357700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46826/0 (socket says 172.21.15.71:46826) 2021-11-20T07:33:22.117 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.110+0000 7fb0f2357700 1 -- 172.21.15.71:0/94838464 learned_addr learned my addr 172.21.15.71:0/94838464 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:22.117 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.110+0000 7fb0f2357700 1 -- 172.21.15.71:0/94838464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0fb5c0 con 0x7fb0ec0faae0 2021-11-20T07:33:22.117 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f2357700 1 --2- 172.21.15.71:0/94838464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb0e8009960 tx=0x7fb0e80092d0).ready entity=mon.0 client_cookie=15afe0c244dad79f server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:22.118 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f1355700 1 -- 172.21.15.71:0/94838464 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e800b9c0 con 0x7fb0ec0faae0 2021-11-20T07:33:22.118 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f1355700 1 -- 172.21.15.71:0/94838464 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e800bb20 con 0x7fb0ec0faae0 2021-11-20T07:33:22.118 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f1355700 1 -- 172.21.15.71:0/94838464 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e800bdf0 con 0x7fb0ec0faae0 2021-11-20T07:33:22.118 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f45bb700 1 -- 172.21.15.71:0/94838464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 msgr2=0x7fb0ec0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:22.118 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f45bb700 1 --2- 172.21.15.71:0/94838464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb0e8009960 tx=0x7fb0e80092d0).stop 2021-11-20T07:33:22.119 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f45bb700 1 -- 172.21.15.71:0/94838464 shutdown_connections 2021-11-20T07:33:22.119 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f45bb700 1 --2- 172.21.15.71:0/94838464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec0faf00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:22.119 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.111+0000 7fb0f45bb700 1 -- 172.21.15.71:0/94838464 >> 172.21.15.71:0/94838464 conn(0x7fb0ec0f6280 msgr2=0x7fb0ec0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:22.119 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 -- 172.21.15.71:0/94838464 shutdown_connections 2021-11-20T07:33:22.120 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 -- 172.21.15.71:0/94838464 wait complete. 2021-11-20T07:33:22.120 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 Processor -- start 2021-11-20T07:33:22.120 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 -- start start 2021-11-20T07:33:22.120 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:22.121 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f45bb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec0fe750 con 0x7fb0ec0faae0 2021-11-20T07:33:22.121 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f2357700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:22.121 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f2357700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46828/0 (socket says 172.21.15.71:46828) 2021-11-20T07:33:22.121 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.112+0000 7fb0f2357700 1 -- 172.21.15.71:0/792798241 learned_addr learned my addr 172.21.15.71:0/792798241 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:22.121 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0f2357700 1 -- 172.21.15.71:0/792798241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e8009570 con 0x7fb0ec0faae0 2021-11-20T07:33:22.122 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0f2357700 1 --2- 172.21.15.71:0/792798241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb0e8009f20 tx=0x7fb0e8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:22.122 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e801c420 con 0x7fb0ec0faae0 2021-11-20T07:33:22.122 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e801c580 con 0x7fb0ec0faae0 2021-11-20T07:33:22.122 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e801c850 con 0x7fb0ec0faae0 2021-11-20T07:33:22.123 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec0fe8f0 con 0x7fb0ec0faae0 2021-11-20T07:33:22.123 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec0fed00 con 0x7fb0ec0faae0 2021-11-20T07:33:22.123 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb0e802c070 con 0x7fb0ec0faae0 2021-11-20T07:33:22.123 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.113+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb0e8025620 con 0x7fb0ec0faae0 2021-11-20T07:33:22.123 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.114+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec059100 con 0x7fb0ec0faae0 2021-11-20T07:33:22.124 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.116+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb0e802a060 con 0x7fb0ec0faae0 2021-11-20T07:33:22.186 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.182+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb0ec0fbf00 con 0x7fb0ec0faae0 2021-11-20T07:33:22.187 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.182+0000 7fb0daffd700 1 -- 172.21.15.71:0/792798241 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fb0e8020070 con 0x7fb0ec0faae0 2021-11-20T07:33:22.187 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: cluster: 2021-11-20T07:33:22.187 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: id: 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:22.187 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-11-20T07:33:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: services: 2021-11-20T07:33:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi071 (age 0.556279s) 2021-11-20T07:33:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mgr: no daemons active 2021-11-20T07:33:22.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-11-20T07:33:22.189 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.189 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: data: 2021-11-20T07:33:22.189 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-11-20T07:33:22.189 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-11-20T07:33:22.190 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-11-20T07:33:22.190 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: pgs: 2021-11-20T07:33:22.190 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.190 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 msgr2=0x7fb0ec1002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:22.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 --2- 172.21.15.71:0/792798241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb0e8009f20 tx=0x7fb0e8017000).stop 2021-11-20T07:33:22.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 shutdown_connections 2021-11-20T07:33:22.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 --2- 172.21.15.71:0/792798241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0ec0faae0 0x7fb0ec1002d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:22.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 >> 172.21.15.71:0/792798241 conn(0x7fb0ec0f6280 msgr2=0x7fb0ec0fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:22.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 shutdown_connections 2021-11-20T07:33:22.192 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:22.183+0000 7fb0f45bb700 1 -- 172.21.15.71:0/792798241 wait complete. 2021-11-20T07:33:22.628 INFO:teuthology.orchestra.run.smithi071.stderr:mon is available 2021-11-20T07:33:22.629 INFO:teuthology.orchestra.run.smithi071.stderr:Assimilating anything we can from ceph.conf... 2021-11-20T07:33:22.629 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071:/var/lib/ceph/mon/ceph-smithi071:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi071/config 2021-11-20T07:33:22.752 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.627515+0000 mon.smithi071 (mon.0) 2 : cluster [INF] mon.smithi071 is new leader, mons smithi071 in quorum (ranks 0) 2021-11-20T07:33:22.753 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.627705+0000 mon.smithi071 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi071=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0]} 2021-11-20T07:33:22.753 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.636437+0000 mon.smithi071 (mon.0) 4 : cluster [DBG] fsmap 2021-11-20T07:33:22.753 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.643308+0000 mon.smithi071 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T07:33:22.753 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: cluster 2021-11-20T07:33:21.643687+0000 mon.smithi071 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T07:33:22.754 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:22 smithi071 conmon[32993]: audit 2021-11-20T07:33:22.183764+0000 mon.smithi071 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.71:0/792798241' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-11-20T07:33:23.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f6530f25700 1 Processor -- start 2021-11-20T07:33:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f6530f25700 1 -- start start 2021-11-20T07:33:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f6530f25700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f6530f25700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c0fc2b0 con 0x7f652c0fb9a0 2021-11-20T07:33:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46830/0 (socket says 172.21.15.71:46830) 2021-11-20T07:33:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.078+0000 7f652b7fe700 1 -- 172.21.15.71:0/4208271659 learned_addr learned my addr 172.21.15.71:0/4208271659 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f652b7fe700 1 -- 172.21.15.71:0/4208271659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c0fc3f0 con 0x7f652c0fb9a0 2021-11-20T07:33:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f652b7fe700 1 --2- 172.21.15.71:0/4208271659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f651c00b5f0 tx=0x7f651c006ee0).ready entity=mon.0 client_cookie=b8708f3569d49058 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f652a7fc700 1 -- 172.21.15.71:0/4208271659 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c00d600 con 0x7f652c0fb9a0 2021-11-20T07:33:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f652a7fc700 1 -- 172.21.15.71:0/4208271659 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c00d760 con 0x7f652c0fb9a0 2021-11-20T07:33:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f652a7fc700 1 -- 172.21.15.71:0/4208271659 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c00da30 con 0x7f652c0fb9a0 2021-11-20T07:33:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f6530f25700 1 -- 172.21.15.71:0/4208271659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 msgr2=0x7f652c0fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.079+0000 7f6530f25700 1 --2- 172.21.15.71:0/4208271659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f651c00b5f0 tx=0x7f651c006ee0).stop 2021-11-20T07:33:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- 172.21.15.71:0/4208271659 shutdown_connections 2021-11-20T07:33:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 --2- 172.21.15.71:0/4208271659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c0fbdc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- 172.21.15.71:0/4208271659 >> 172.21.15.71:0/4208271659 conn(0x7f652c0f71e0 msgr2=0x7f652c0f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- 172.21.15.71:0/4208271659 shutdown_connections 2021-11-20T07:33:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- 172.21.15.71:0/4208271659 wait complete. 2021-11-20T07:33:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 Processor -- start 2021-11-20T07:33:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- start start 2021-11-20T07:33:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.080+0000 7f6530f25700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c10a500 con 0x7f652c0fb9a0 2021-11-20T07:33:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f652b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46832/0 (socket says 172.21.15.71:46832) 2021-11-20T07:33:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f652b7fe700 1 -- 172.21.15.71:0/963881221 learned_addr learned my addr 172.21.15.71:0/963881221 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f652b7fe700 1 -- 172.21.15.71:0/963881221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c00b200 con 0x7f652c0fb9a0 2021-11-20T07:33:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f652b7fe700 1 --2- 172.21.15.71:0/963881221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f651c006c70 tx=0x7f651c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c011b40 con 0x7f652c0fb9a0 2021-11-20T07:33:23.092 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c011ca0 con 0x7f652c0fb9a0 2021-11-20T07:33:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c01d430 con 0x7f652c0fb9a0 2021-11-20T07:33:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c10a700 con 0x7f652c0fb9a0 2021-11-20T07:33:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c10ab10 con 0x7f652c0fb9a0 2021-11-20T07:33:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f651c00f070 con 0x7f652c0fb9a0 2021-11-20T07:33:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.081+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f651c027460 con 0x7f652c0fb9a0 2021-11-20T07:33:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.082+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652c045ad0 con 0x7f652c0fb9a0 2021-11-20T07:33:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.084+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f651c016070 con 0x7f652c0fb9a0 2021-11-20T07:33:23.150 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.148+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f652c0fcce0 con 0x7f652c0fb9a0 2021-11-20T07:33:23.156 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.151+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f651c01d620 con 0x7f652c0fb9a0 2021-11-20T07:33:23.156 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.151+0000 7f6513fff700 1 -- 172.21.15.71:0/963881221 <== mon.0 v2:172.21.15.71:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f651c030880 con 0x7f652c0fb9a0 2021-11-20T07:33:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: [global] 2021-11-20T07:33:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: fsid = 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:23.157 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.71:3300,v1:172.21.15.71:6789] 2021-11-20T07:33:23.158 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-11-20T07:33:23.158 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-11-20T07:33:23.158 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-11-20T07:33:23.159 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-11-20T07:33:23.159 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.159 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: [mgr] 2021-11-20T07:33:23.159 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-11-20T07:33:23.160 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.160 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: [osd] 2021-11-20T07:33:23.160 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2021-11-20T07:33:23.160 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2021-11-20T07:33:23.160 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2021-11-20T07:33:23.161 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-11-20T07:33:23.161 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2021-11-20T07:33:23.161 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-11-20T07:33:23.161 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 msgr2=0x7f652c10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:23.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 --2- 172.21.15.71:0/963881221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f651c006c70 tx=0x7f651c016000).stop 2021-11-20T07:33:23.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 shutdown_connections 2021-11-20T07:33:23.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 --2- 172.21.15.71:0/963881221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f652c0fb9a0 0x7f652c10a010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:23.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 >> 172.21.15.71:0/963881221 conn(0x7f652c0f71e0 msgr2=0x7f652c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:23.163 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 shutdown_connections 2021-11-20T07:33:23.163 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.153+0000 7f6530f25700 1 -- 172.21.15.71:0/963881221 wait complete. 2021-11-20T07:33:23.465 INFO:teuthology.orchestra.run.smithi071.stderr:Generating new minimal ceph.conf... 2021-11-20T07:33:23.465 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z -v /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071:/var/lib/ceph/mon/ceph-smithi071:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi071/config 2021-11-20T07:33:23.900 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:23 smithi071 conmon[32993]: audit 2021-11-20T07:33:23.149948+0000 mon.smithi071 (mon.0 2021-11-20T07:33:23.900 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:23 smithi071 conmon[32993]: ) 8 : audit [INF] from='client.? 172.21.15.71:0/963881221' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T07:33:23.900 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:23 smithi071 conmon[32993]: audit 2021-11-20T07:33:23.153053+0000 mon.smithi071 (mon.0) 9 : audit [INF] from='client.? 172.21.15.71:0/963881221' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T07:33:23.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.898+0000 7faf10a71700 1 Processor -- start 2021-11-20T07:33:23.904 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.898+0000 7faf10a71700 1 -- start start 2021-11-20T07:33:23.904 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.898+0000 7faf10a71700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:23.904 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.898+0000 7faf10a71700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0720f0 con 0x7faf0c071cd0 2021-11-20T07:33:23.905 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:23.905 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46834/0 (socket says 172.21.15.71:46834) 2021-11-20T07:33:23.905 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0a59c700 1 -- 172.21.15.71:0/1767363296 learned_addr learned my addr 172.21.15.71:0/1767363296 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:23.905 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0a59c700 1 -- 172.21.15.71:0/1767363296 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c070760 con 0x7faf0c071cd0 2021-11-20T07:33:23.906 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0a59c700 1 --2- 172.21.15.71:0/1767363296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7faefc009960 tx=0x7faefc0092d0).ready entity=mon.0 client_cookie=388f5ed536f96788 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:23.906 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0959a700 1 -- 172.21.15.71:0/1767363296 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faefc00b9c0 con 0x7faf0c071cd0 2021-11-20T07:33:23.906 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0959a700 1 -- 172.21.15.71:0/1767363296 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faefc011400 con 0x7faf0c071cd0 2021-11-20T07:33:23.906 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.899+0000 7faf0959a700 1 -- 172.21.15.71:0/1767363296 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faefc0103a0 con 0x7faf0c071cd0 2021-11-20T07:33:23.907 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 -- 172.21.15.71:0/1767363296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 msgr2=0x7faf0c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:23.907 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 --2- 172.21.15.71:0/1767363296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7faefc009960 tx=0x7faefc0092d0).stop 2021-11-20T07:33:23.907 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 -- 172.21.15.71:0/1767363296 shutdown_connections 2021-11-20T07:33:23.907 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 --2- 172.21.15.71:0/1767363296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c070270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:23.907 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 -- 172.21.15.71:0/1767363296 >> 172.21.15.71:0/1767363296 conn(0x7faf0c0f73f0 msgr2=0x7faf0c0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:23.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 -- 172.21.15.71:0/1767363296 shutdown_connections 2021-11-20T07:33:23.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.900+0000 7faf10a71700 1 -- 172.21.15.71:0/1767363296 wait complete. 2021-11-20T07:33:23.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf10a71700 1 Processor -- start 2021-11-20T07:33:23.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf10a71700 1 -- start start 2021-11-20T07:33:23.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf10a71700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:23.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf10a71700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c10d080 con 0x7faf0c071cd0 2021-11-20T07:33:23.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:23.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf0a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46836/0 (socket says 172.21.15.71:46836) 2021-11-20T07:33:23.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf0a59c700 1 -- 172.21.15.71:0/2525706956 learned_addr learned my addr 172.21.15.71:0/2525706956 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:23.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf0a59c700 1 -- 172.21.15.71:0/2525706956 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc009570 con 0x7faf0c071cd0 2021-11-20T07:33:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf0a59c700 1 --2- 172.21.15.71:0/2525706956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faefc011c70 tx=0x7faefc0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faefc010870 con 0x7faf0c071cd0 2021-11-20T07:33:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faefc010e40 con 0x7faf0c071cd0 2021-11-20T07:33:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.901+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0c10d280 con 0x7faf0c071cd0 2021-11-20T07:33:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.902+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faefc01ae30 con 0x7faf0c071cd0 2021-11-20T07:33:23.912 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.902+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c10d690 con 0x7faf0c071cd0 2021-11-20T07:33:23.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.902+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7faefc02a070 con 0x7faf0c071cd0 2021-11-20T07:33:23.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.902+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7faefc022730 con 0x7faf0c071cd0 2021-11-20T07:33:23.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.903+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c045ad0 con 0x7faf0c071cd0 2021-11-20T07:33:23.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.905+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7faefc028060 con 0x7faf0c071cd0 2021-11-20T07:33:23.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.970+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7faf0c101d90 con 0x7faf0c071cd0 2021-11-20T07:33:23.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.970+0000 7faf02ffd700 1 -- 172.21.15.71:0/2525706956 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7faefc031090 con 0x7faf0c071cd0 2021-11-20T07:33:23.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 msgr2=0x7faf0c10cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:23.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 --2- 172.21.15.71:0/2525706956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faefc011c70 tx=0x7faefc0092d0).stop 2021-11-20T07:33:23.974 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 shutdown_connections 2021-11-20T07:33:23.974 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 --2- 172.21.15.71:0/2525706956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf0c071cd0 0x7faf0c10cb90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:23.974 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 >> 172.21.15.71:0/2525706956 conn(0x7faf0c0f73f0 msgr2=0x7faf0c0f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:23.975 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 shutdown_connections 2021-11-20T07:33:23.975 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:23.971+0000 7faf10a71700 1 -- 172.21.15.71:0/2525706956 wait complete. 2021-11-20T07:33:24.402 INFO:teuthology.orchestra.run.smithi071.stderr:Restarting the monitor... 2021-11-20T07:33:24.402 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl restart ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:33:24.698 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 systemd[1]: Stopping Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:33:24.699 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 conmon[32993]: debug 2021-11-20T07:33:24.477+0000 7f03bc6a3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi071 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T07:33:24.699 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 conmon[32993]: debug 2021-11-20T07:33:24.700 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 conmon[32993]: 2021-11-20T07:33:24.477+0000 7f03bc6a3700 -1 mon.smithi071@0(leader) e1 *** Got Signal Terminated *** 2021-11-20T07:33:25.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 podman[33478]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-mon.smithi071 2021-11-20T07:33:25.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 systemd[1]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service: Succeeded. 2021-11-20T07:33:25.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 systemd[1]: Stopped Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c. 2021-11-20T07:33:25.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:24 smithi071 systemd[1]: Starting Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:33:25.366 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 podman[33647]: 2021-11-20T07:33:25.366 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.283+0000 7f9c2d545700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T07:33:25.367 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.283+0000 7f9c2d545700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 8 2021-11-20T07:33:25.373 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.283+0000 7f9c2d545700 0 pidfile_write: ignore empty --pid-file 2021-11-20T07:33:25.374 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 bash[33569]: 95ec9c4b46c6f8c1a206e3b5ec2fbb3196ac69a89e29d4d05a64423fc4a216db 2021-11-20T07:33:25.374 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.287+0000 7f9c2d545700 0 load: jerasure load: lrc load: isa 2021-11-20T07:33:25.374 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T07:33:25.374 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.375 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T07:33:25.375 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Compile date May 13 2021 2021-11-20T07:33:25.375 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: DB SUMMARY 2021-11-20T07:33:25.375 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.376 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T07:33:25.376 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.376 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T07:33:25.376 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.376 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-11-20T07:33:25.377 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.377 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi071/store.db dir, Total Num: 1, files: 000004.sst 2021-11-20T07:33:25.377 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.378 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi071/store.db: 000006.log size: 92460 ; 2021-11-20T07:33:25.378 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.378 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T07:33:25.378 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T07:33:25.379 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T07:33:25.379 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.env: 0x560764b911c0 2021-11-20T07:33:25.379 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.fs: Posix File System 2021-11-20T07:33:25.379 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.info_log: 0x5607659e1960 2021-11-20T07:33:25.379 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T07:33:25.380 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.statistics: (nil) 2021-11-20T07:33:25.380 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.use_fsync: 0 2021-11-20T07:33:25.380 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T07:33:25.380 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T07:33:25.380 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T07:33:25.381 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T07:33:25.381 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T07:33:25.381 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T07:33:25.381 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T07:33:25.382 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T07:33:25.382 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.382 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T07:33:25.382 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T07:33:25.382 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T07:33:25.383 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.db_log_dir: 2021-11-20T07:33:25.383 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi071/store.db 2021-11-20T07:33:25.383 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T07:33:25.383 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T07:33:25.384 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T07:33:25.384 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T07:33:25.384 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T07:33:25.384 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T07:33:25.384 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T07:33:25.385 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T07:33:25.385 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T07:33:25.385 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T07:33:25.385 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.write_buffer_manager: 0x560766656030 2021-11-20T07:33:25.385 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T07:33:25.386 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T07:33:25.386 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T07:33:25.386 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T07:33:25.386 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T07:33:25.387 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T07:33:25.387 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T07:33:25.387 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T07:33:25.387 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T07:33:25.387 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.unordered_write: 0 2021-11-20T07:33:25.388 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T07:33:25.388 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T07:33:25.388 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T07:33:25.388 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T07:33:25.389 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.row_cache: None 2021-11-20T07:33:25.389 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.wal_filter: None 2021-11-20T07:33:25.389 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T07:33:25.389 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T07:33:25.390 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T07:33:25.390 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T07:33:25.390 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T07:33:25.391 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T07:33:25.391 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T07:33:25.391 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T07:33:25.391 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T07:33:25.392 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T07:33:25.392 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T07:33:25.392 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T07:33:25.392 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T07:33:25.393 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T07:33:25.393 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T07:33:25.393 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T07:33:25.393 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T07:33:25.394 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.394 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T07:33:25.394 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T07:33:25.394 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T07:33:25.394 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T07:33:25.395 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_open_files: -1 2021-11-20T07:33:25.395 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T07:33:25.395 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T07:33:25.395 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T07:33:25.396 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T07:33:25.396 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Compression algorithms supported: 2021-11-20T07:33:25.396 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T07:33:25.396 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kZSTD supported: 0 2021-11-20T07:33:25.396 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.397 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T07:33:25.397 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T07:33:25.397 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T07:33:25.397 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T07:33:25.398 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T07:33:25.398 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T07:33:25.398 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T07:33:25.398 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi071/store.db/MANIFEST-000005 2021-11-20T07:33:25.398 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.399 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T07:33:25.399 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.399 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T07:33:25.399 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.merge_operator: 2021-11-20T07:33:25.400 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_filter: None 2021-11-20T07:33:25.400 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T07:33:25.400 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.400 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T07:33:25.401 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T07:33:25.401 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607658f8128) 2021-11-20T07:33:25.401 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cache_index_and_filter_blocks: 1 2021-11-20T07:33:25.401 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T07:33:25.401 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T07:33:25.402 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: pin_top_level_index_and_filter: 1 2021-11-20T07:33:25.402 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: index_type: 0 2021-11-20T07:33:25.402 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: data_block_index_type: 0 2021-11-20T07:33:25.402 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: index_shortening: 1 2021-11-20T07:33:25.403 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T07:33:25.403 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: hash_index_allow_collision: 1 2021-11-20T07:33:25.403 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: checksum: 1 2021-11-20T07:33:25.403 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: no_block_cache: 0 2021-11-20T07:33:25.403 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_cache: 0x56076592ef10 2021-11-20T07:33:25.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_cache_name: BinnedLRUCache 2021-11-20T07:33:25.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_cache_options: 2021-11-20T07:33:25.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: capacity : 536870912 2021-11-20T07:33:25.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: num_shard_bits : 4 2021-11-20T07:33:25.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: strict_capacity_limit : 0 2021-11-20T07:33:25.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: high_pri_pool_ratio: 0.000 2021-11-20T07:33:25.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_cache_compressed: (nil) 2021-11-20T07:33:25.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: persistent_cache: (nil) 2021-11-20T07:33:25.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_size: 4096 2021-11-20T07:33:25.406 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_size_deviation: 10 2021-11-20T07:33:25.406 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_restart_interval: 16 2021-11-20T07:33:25.406 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: index_block_restart_interval: 1 2021-11-20T07:33:25.406 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: metadata_block_size: 4096 2021-11-20T07:33:25.407 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: partition_filters: 0 2021-11-20T07:33:25.407 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: use_delta_encoding: 1 2021-11-20T07:33:25.407 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T07:33:25.407 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: whole_key_filtering: 1 2021-11-20T07:33:25.407 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: verify_compression: 0 2021-11-20T07:33:25.408 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: read_amp_bytes_per_bit: 0 2021-11-20T07:33:25.408 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: format_version: 2 2021-11-20T07:33:25.408 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: enable_index_compression: 1 2021-11-20T07:33:25.408 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: block_align: 0 2021-11-20T07:33:25.408 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.409 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T07:33:25.409 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T07:33:25.409 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression: NoCompression 2021-11-20T07:33:25.409 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T07:33:25.410 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T07:33:25.410 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T07:33:25.410 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.num_levels: 7 2021-11-20T07:33:25.410 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T07:33:25.411 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T07:33:25.411 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T07:33:25.411 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T07:33:25.411 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T07:33:25.412 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T07:33:25.412 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T07:33:25.412 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:33:25.412 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T07:33:25.412 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T07:33:25.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T07:33:25.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T07:33:25.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T07:33:25.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:33:25.414 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T07:33:25.414 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T07:33:25.414 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T07:33:25.414 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T07:33:25.414 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T07:33:25.415 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T07:33:25.415 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T07:33:25.415 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T07:33:25.415 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T07:33:25.415 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T07:33:25.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T07:33:25.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T07:33:25.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T07:33:25.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T07:33:25.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T07:33:25.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T07:33:25.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T07:33:25.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T07:33:25.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T07:33:25.418 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T07:33:25.418 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T07:33:25.418 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T07:33:25.418 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T07:33:25.419 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T07:33:25.419 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T07:33:25.419 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T07:33:25.419 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T07:33:25.419 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T07:33:25.420 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T07:33:25.420 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T07:33:25.420 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T07:33:25.420 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T07:33:25.421 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T07:33:25.421 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.421 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T07:33:25.421 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T07:33:25.421 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T07:33:25.422 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T07:33:25.422 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T07:33:25.422 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T07:33:25.422 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T07:33:25.423 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T07:33:25.423 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T07:33:25.423 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T07:33:25.423 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T07:33:25.423 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T07:33:25.424 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.ttl: 2592000 2021-11-20T07:33:25.424 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.288+0000 7f9c2d545700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T07:33:25.424 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.289+0000 7f9c2d545700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi071/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T07:33:25.424 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.425 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.289+0000 7f9c2d545700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-11-20T07:33:25.425 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.425 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.289+0000 7f9c2d545700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393605290520, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-11-20T07:33:25.425 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.425 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.289+0000 7f9c2d545700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-11-20T07:33:25.426 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.289+0000 7f9c2d545700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T07:33:25.426 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.290+0000 7f9c2d545700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393605291717, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88260, "table_properties": {"data_size": 86387, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79601, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637393605, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T07:33:25.426 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.290+0000 7f9c2d545700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-11-20T07:33:25.426 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.427 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.291+0000 7f9c2d545700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393605292603, "job": 1, "event": "recovery_finished"} 2021-11-20T07:33:25.428 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.292+0000 7f9c2d545700 4 rocksdb: DB pointer 0x5607659f3800 2021-11-20T07:33:25.428 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.292+0000 7f9c1435d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T07:33:25.428 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.292+0000 7f9c1435d700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T07:33:25.428 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** DB Stats ** 2021-11-20T07:33:25.429 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:25.429 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T07:33:25.429 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T07:33:25.429 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:33:25.430 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T07:33:25.430 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T07:33:25.430 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:33:25.430 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.430 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** Compaction Stats [default] ** 2021-11-20T07:33:25.431 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:25.431 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:25.431 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: L0 2/0 87.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.431 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Sum 2/0 87.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** Compaction Stats [default] ** 2021-11-20T07:33:25.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:25.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:25.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:25.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:25.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:25.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:33:25.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:33:25.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:33:25.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Cumulative compaction: 0.00 GB write, 21.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:25.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Interval compaction: 0.00 GB write, 21.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:25.435 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:33:25.435 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.435 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:33:25.435 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.436 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** Compaction Stats [default] ** 2021-11-20T07:33:25.436 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:25.436 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:25.436 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: L0 2/0 87.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.437 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Sum 2/0 87.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.437 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:33:25.437 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.437 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** Compaction Stats [default] ** 2021-11-20T07:33:25.438 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:33:25.438 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:33:25.438 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.0 0.00 0.00 1 0.001 0 0 2021-11-20T07:33:25.438 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:33:25.438 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:25.439 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:33:25.439 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:33:25.439 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:33:25.439 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:33:25.440 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Cumulative compaction: 0.00 GB write, 21.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:25.440 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:33:25.440 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:33:25.440 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.440 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:33:25.441 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.441 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.293+0000 7f9c2d545700 0 starting mon.smithi071 rank 0 at public addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] at bind addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi071 fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:25.441 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.294+0000 7f9c2d545700 1 mon.smithi071@-1(???) e1 preinit fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:25.441 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).mds e1 new map 2021-11-20T07:33:25.442 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).mds e1 print_map 2021-11-20T07:33:25.442 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: e1 2021-11-20T07:33:25.442 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T07:33:25.442 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:33:25.442 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: legacy client fscid: -1 2021-11-20T07:33:25.443 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.443 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: No filesystems configured 2021-11-20T07:33:25.443 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.443 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T07:33:25.444 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.444 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:33:25.444 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:33:25.444 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 0 mon.smithi071@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:33:25.444 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: audit 2021-11-20T07:33:25.445 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T 2021-11-20T07:33:25.445 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 07:33:23.149948+0000 2021-11-20T07:33:25.445 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: mon.smithi071 (mon.0) 8 : 2021-11-20T07:33:25.445 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: audit [INF] from='client.? 172.21.15.71:0/963881221' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T07:33:25.446 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: audit 2021-11-20T07:33:25.446 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33 2021-11-20T07:33:25.446 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: :23.153053+0000 mon.smithi071 ( 2021-11-20T07:33:25.446 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: mon.0) 9 : audit [INF] 2021-11-20T07:33:25.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/963881221' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T07:33:25.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: debug 2021-11-20T07:33:25.295+0000 7f9c2d545700 1 mon.smithi071@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-11-20T07:33:25.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cluster 2021-11-20T07:33:25.312838+0000 mon.smithi071 (mon.0) 1 : cluster [INF] mon.smithi071 is new leader, mons smithi071 in quorum (ranks 0) 2021-11-20T07:33:25.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cluster 2021-11-20T07:33:25.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: 2021-11-20T07:33:25.312895+0000 mon.smithi071 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi071=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0]} 2021-11-20T07:33:25.448 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cluster 2021-11-20T07:33:25.312927+0000 mon.smithi071 (mon.0) 3 : cluster [DBG] fsmap 2021-11-20T07:33:25.448 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cluster 2021-11-20T07:33:25.312950+0000 mon.smithi071 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T07:33:25.448 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 conmon[33687]: cluster 2021-11-20T07:33:25.313285+0000 mon.smithi071 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T07:33:25.449 INFO:teuthology.orchestra.run.smithi071.stderr:Setting mon public_network to 172.21.0.0/20 2021-11-20T07:33:25.449 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2021-11-20T07:33:25.628 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:25 smithi071 systemd[1]: Started Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c. 2021-11-20T07:33:25.855 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.845+0000 7f716f59e700 1 Processor -- start 2021-11-20T07:33:25.856 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.845+0000 7f716f59e700 1 -- start start 2021-11-20T07:33:25.856 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716f59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:25.857 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716f59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700fc8d0 con 0x7f71700fbfc0 2021-11-20T07:33:25.857 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:25.858 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46838/0 (socket says 172.21.15.71:46838) 2021-11-20T07:33:25.858 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716e59c700 1 -- 172.21.15.71:0/650390372 learned_addr learned my addr 172.21.15.71:0/650390372 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716e59c700 1 -- 172.21.15.71:0/650390372 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700fca10 con 0x7f71700fbfc0 2021-11-20T07:33:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.846+0000 7f716e59c700 1 --2- 172.21.15.71:0/650390372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f716000b5f0 tx=0x7f7160006ee0).ready entity=mon.0 client_cookie=e60f4899d29546fd server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.847+0000 7f716d59a700 1 -- 172.21.15.71:0/650390372 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f716000d600 con 0x7f71700fbfc0 2021-11-20T07:33:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.847+0000 7f716d59a700 1 -- 172.21.15.71:0/650390372 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f716000dbd0 con 0x7f71700fbfc0 2021-11-20T07:33:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.847+0000 7f716d59a700 1 -- 172.21.15.71:0/650390372 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7160011c30 con 0x7f71700fbfc0 2021-11-20T07:33:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.847+0000 7f716f59e700 1 -- 172.21.15.71:0/650390372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 msgr2=0x7f71700fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.847+0000 7f716f59e700 1 --2- 172.21.15.71:0/650390372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f716000b5f0 tx=0x7f7160006ee0).stop 2021-11-20T07:33:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- 172.21.15.71:0/650390372 shutdown_connections 2021-11-20T07:33:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 --2- 172.21.15.71:0/650390372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f71700fc3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- 172.21.15.71:0/650390372 >> 172.21.15.71:0/650390372 conn(0x7f71700f5870 msgr2=0x7f71700f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:25.863 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- 172.21.15.71:0/650390372 shutdown_connections 2021-11-20T07:33:25.863 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- 172.21.15.71:0/650390372 wait complete. 2021-11-20T07:33:25.864 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 Processor -- start 2021-11-20T07:33:25.864 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- start start 2021-11-20T07:33:25.865 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:25.865 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.848+0000 7f716f59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717010cee0 con 0x7f71700fbfc0 2021-11-20T07:33:25.865 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:25.866 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46840/0 (socket says 172.21.15.71:46840) 2021-11-20T07:33:25.866 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716e59c700 1 -- 172.21.15.71:0/3886045407 learned_addr learned my addr 172.21.15.71:0/3886045407 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:25.867 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716e59c700 1 -- 172.21.15.71:0/3886045407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716000b200 con 0x7f71700fbfc0 2021-11-20T07:33:25.867 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716e59c700 1 --2- 172.21.15.71:0/3886045407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f716000b1d0 tx=0x7f7160006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:25.868 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f716001b410 con 0x7f71700fbfc0 2021-11-20T07:33:25.868 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f716001b9e0 con 0x7f71700fbfc0 2021-11-20T07:33:25.868 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f716001a7f0 con 0x7f71700fbfc0 2021-11-20T07:33:25.869 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717010d0e0 con 0x7f71700fbfc0 2021-11-20T07:33:25.869 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717010d4f0 con 0x7f71700fbfc0 2021-11-20T07:33:25.870 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f716000f070 con 0x7f71700fbfc0 2021-11-20T07:33:25.870 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.849+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7160025b30 con 0x7f71700fbfc0 2021-11-20T07:33:25.871 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.850+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7170045ad0 con 0x7f71700fbfc0 2021-11-20T07:33:25.871 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.852+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7160016070 con 0x7f71700fbfc0 2021-11-20T07:33:25.917 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.916+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f71701068e0 con 0x7f71700fbfc0 2021-11-20T07:33:25.920 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.918+0000 7f715effd700 1 -- 172.21.15.71:0/3886045407 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7160025d70 con 0x7f71700fbfc0 2021-11-20T07:33:25.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 msgr2=0x7f717010c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:25.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 --2- 172.21.15.71:0/3886045407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f716000b1d0 tx=0x7f7160006ee0).stop 2021-11-20T07:33:25.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 shutdown_connections 2021-11-20T07:33:25.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 --2- 172.21.15.71:0/3886045407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71700fbfc0 0x7f717010c9f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:25.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 >> 172.21.15.71:0/3886045407 conn(0x7f71700f5870 msgr2=0x7f71700f73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:25.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.919+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 shutdown_connections 2021-11-20T07:33:25.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:25.920+0000 7f716f59e700 1 -- 172.21.15.71:0/3886045407 wait complete. 2021-11-20T07:33:26.321 INFO:teuthology.orchestra.run.smithi071.stderr:Wrote config to /etc/ceph/ceph.conf 2021-11-20T07:33:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:33:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:Creating mgr... 2021-11-20T07:33:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:Verifying port 9283 ... 2021-11-20T07:33:26.324 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl daemon-reload 2021-11-20T07:33:26.410 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn 2021-11-20T07:33:26.431 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl reset-failed ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn 2021-11-20T07:33:26.436 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Failed to reset failed state of unit ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn.service: Unit ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn.service not loaded. 2021-11-20T07:33:26.437 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl enable ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn 2021-11-20T07:33:26.442 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: Created symlink /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c.target.wants/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn.service → /etc/systemd/system/ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@.service. 2021-11-20T07:33:26.518 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl start ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mgr.smithi071.ynqmwn 2021-11-20T07:33:27.046 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T07:33:27.055 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: disabled 2021-11-20T07:33:27.055 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T07:33:27.062 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:33:27.063 INFO:teuthology.orchestra.run.smithi071.stderr:firewalld.service is not enabled 2021-11-20T07:33:27.063 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T07:33:27.063 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T07:33:27.064 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T07:33:27.070 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: disabled 2021-11-20T07:33:27.070 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T07:33:27.077 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:33:27.078 INFO:teuthology.orchestra.run.smithi071.stderr:firewalld.service is not enabled 2021-11-20T07:33:27.078 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-11-20T07:33:27.079 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mgr to start... 2021-11-20T07:33:27.079 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mgr... 2021-11-20T07:33:27.079 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T07:33:27.217 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:26 smithi071 conmon[33687]: audit 2021-11-20T07:33:25. 2021-11-20T07:33:27.217 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:26 smithi071 conmon[33687]: 919898+0000 mon.smithi071 (mon.0) 6 : audit [INF] from='client.? 172.21.15.71:0/3886045407' entity='client.admin' 2021-11-20T07:33:27.534 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.522+0000 7f5ce1210700 1 Processor -- start 2021-11-20T07:33:27.534 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.522+0000 7f5ce1210700 1 -- start start 2021-11-20T07:33:27.535 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.522+0000 7f5ce1210700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:27.535 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.522+0000 7f5ce1210700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0fb1b0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.535 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:27.536 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46846/0 (socket says 172.21.15.71:46846) 2021-11-20T07:33:27.536 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdbfff700 1 -- 172.21.15.71:0/2835088503 learned_addr learned my addr 172.21.15.71:0/2835088503 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:27.537 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdbfff700 1 -- 172.21.15.71:0/2835088503 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0fb2f0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.537 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdbfff700 1 --2- 172.21.15.71:0/2835088503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5cd000b870 tx=0x7f5cd0006f70).ready entity=mon.0 client_cookie=e697d15a580750ae server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:27.537 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdaffd700 1 -- 172.21.15.71:0/2835088503 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000da20 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdaffd700 1 -- 172.21.15.71:0/2835088503 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cd0010470 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.523+0000 7f5cdaffd700 1 -- 172.21.15.71:0/2835088503 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000f5b0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.524+0000 7f5ce1210700 1 -- 172.21.15.71:0/2835088503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 msgr2=0x7f5cdc0facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.524+0000 7f5ce1210700 1 --2- 172.21.15.71:0/2835088503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5cd000b870 tx=0x7f5cd0006f70).stop 2021-11-20T07:33:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.524+0000 7f5ce1210700 1 -- 172.21.15.71:0/2835088503 shutdown_connections 2021-11-20T07:33:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.524+0000 7f5ce1210700 1 --2- 172.21.15.71:0/2835088503 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc0facc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.524+0000 7f5ce1210700 1 -- 172.21.15.71:0/2835088503 >> 172.21.15.71:0/2835088503 conn(0x7f5cdc0f60e0 msgr2=0x7f5cdc0f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.525+0000 7f5ce1210700 1 -- 172.21.15.71:0/2835088503 shutdown_connections 2021-11-20T07:33:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.525+0000 7f5ce1210700 1 -- 172.21.15.71:0/2835088503 wait complete. 2021-11-20T07:33:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.525+0000 7f5ce1210700 1 Processor -- start 2021-11-20T07:33:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.525+0000 7f5ce1210700 1 -- start start 2021-11-20T07:33:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5ce1210700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5ce1210700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc108a00 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46848/0 (socket says 172.21.15.71:46848) 2021-11-20T07:33:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cdbfff700 1 -- 172.21.15.71:0/439581753 learned_addr learned my addr 172.21.15.71:0/439581753 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cdbfff700 1 -- 172.21.15.71:0/439581753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd000b480 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cdbfff700 1 --2- 172.21.15.71:0/439581753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5cd0006a40 tx=0x7f5cd0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.526+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000de00 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cd000fef0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd0019e90 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc108c00 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc109010 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5cd0016030 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.544 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.527+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5cd0023760 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.544 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.528+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc045ad0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.544 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.530+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5cd00429c0 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.608 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.599+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5cdc0fc760 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.599+0000 7f5cd8ff9700 1 -- 172.21.15.71:0/439581753 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5cd0031020 con 0x7f5cdc0fa8a0 2021-11-20T07:33:27.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:33:27.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsid": "16910ea2-49d4-11ec-8c2c-001a4aab830c", 2021-11-20T07:33:27.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "health": { 2021-11-20T07:33:27.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T07:33:27.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T07:33:27.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T07:33:27.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T07:33:27.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T07:33:27.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 0 2021-11-20T07:33:27.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:27.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T07:33:27.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "smithi071" 2021-11-20T07:33:27.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:27.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-11-20T07:33:27.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T07:33:27.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:27.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T07:33:27.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T07:33:27.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T07:33:27.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:27.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T07:33:27.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T07:33:27.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T07:33:27.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T07:33:27.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T07:33:27.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T07:33:27.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T07:33:27.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T07:33:27.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T07:33:27.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T07:33:27.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T07:33:27.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T07:33:27.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T07:33:27.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T07:33:27.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T07:33:27.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T07:33:27.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:27.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T07:33:27.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T07:33:27.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T07:33:27.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "available": false, 2021-11-20T07:33:27.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T07:33:27.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T07:33:27.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "iostat", 2021-11-20T07:33:27.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "restful" 2021-11-20T07:33:27.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:27.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:27.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T07:33:27.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:27.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modified": "2021-11-20T07:33:21.631103+0000", 2021-11-20T07:33:27.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:27.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:27.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T07:33:27.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:33:27.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 msgr2=0x7f5cdc108510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:27.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 --2- 172.21.15.71:0/439581753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5cd0006a40 tx=0x7f5cd0006f70).stop 2021-11-20T07:33:27.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 shutdown_connections 2021-11-20T07:33:27.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 --2- 172.21.15.71:0/439581753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cdc0fa8a0 0x7f5cdc108510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:27.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 >> 172.21.15.71:0/439581753 conn(0x7f5cdc0f60e0 msgr2=0x7f5cdc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:27.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 shutdown_connections 2021-11-20T07:33:27.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:27.600+0000 7f5ce1210700 1 -- 172.21.15.71:0/439581753 wait complete. 2021-11-20T07:33:27.918 INFO:teuthology.orchestra.run.smithi071.stderr:mgr not available, waiting (1/15)... 2021-11-20T07:33:28.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:27 smithi071 conmon[33687]: audit 2021-11-20T07:33:27.600596+0000 mon.smithi071 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.71:0/439581753' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T07:33:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T07:33:30.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.385+0000 7f742a1e4700 1 Processor -- start 2021-11-20T07:33:30.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.385+0000 7f742a1e4700 1 -- start start 2021-11-20T07:33:30.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f742a1e4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:30.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f742a1e4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74240fca80 con 0x7f74240fc0e0 2021-11-20T07:33:30.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f74237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:30.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f74237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46850/0 (socket says 172.21.15.71:46850) 2021-11-20T07:33:30.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f74237fe700 1 -- 172.21.15.71:0/2780105574 learned_addr learned my addr 172.21.15.71:0/2780105574 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:30.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f74237fe700 1 -- 172.21.15.71:0/2780105574 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74240fcbc0 con 0x7f74240fc0e0 2021-11-20T07:33:30.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.386+0000 7f74237fe700 1 --2- 172.21.15.71:0/2780105574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f740c00b5f0 tx=0x7f740c006ee0).ready entity=mon.0 client_cookie=3d4d0ddf26602349 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:30.392 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f74227fc700 1 -- 172.21.15.71:0/2780105574 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c00d600 con 0x7f74240fc0e0 2021-11-20T07:33:30.392 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f74227fc700 1 -- 172.21.15.71:0/2780105574 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f740c00dbd0 con 0x7f74240fc0e0 2021-11-20T07:33:30.392 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f74227fc700 1 -- 172.21.15.71:0/2780105574 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c011c00 con 0x7f74240fc0e0 2021-11-20T07:33:30.392 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 -- 172.21.15.71:0/2780105574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 msgr2=0x7f74240fc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:30.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 --2- 172.21.15.71:0/2780105574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f740c00b5f0 tx=0x7f740c006ee0).stop 2021-11-20T07:33:30.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 -- 172.21.15.71:0/2780105574 shutdown_connections 2021-11-20T07:33:30.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 --2- 172.21.15.71:0/2780105574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f74240fc500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:30.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 -- 172.21.15.71:0/2780105574 >> 172.21.15.71:0/2780105574 conn(0x7f74240f59e0 msgr2=0x7f74240f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:30.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 -- 172.21.15.71:0/2780105574 shutdown_connections 2021-11-20T07:33:30.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.387+0000 7f742a1e4700 1 -- 172.21.15.71:0/2780105574 wait complete. 2021-11-20T07:33:30.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f742a1e4700 1 Processor -- start 2021-11-20T07:33:30.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f742a1e4700 1 -- start start 2021-11-20T07:33:30.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f742a1e4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:30.395 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f742a1e4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74240faa70 con 0x7f74240fc0e0 2021-11-20T07:33:30.395 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f74237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:30.396 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f74237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46852/0 (socket says 172.21.15.71:46852) 2021-11-20T07:33:30.396 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.388+0000 7f74237fe700 1 -- 172.21.15.71:0/1383834089 learned_addr learned my addr 172.21.15.71:0/1383834089 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:30.396 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f74237fe700 1 -- 172.21.15.71:0/1383834089 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c00b200 con 0x7f74240fc0e0 2021-11-20T07:33:30.396 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f74237fe700 1 --2- 172.21.15.71:0/1383834089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f740c00b1d0 tx=0x7f740c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:30.397 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c011a20 con 0x7f74240fc0e0 2021-11-20T07:33:30.397 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f740c01b880 con 0x7f74240fc0e0 2021-11-20T07:33:30.397 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c01a700 con 0x7f74240fc0e0 2021-11-20T07:33:30.397 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74240fac70 con 0x7f74240fc0e0 2021-11-20T07:33:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74241096c0 con 0x7f74240fc0e0 2021-11-20T07:33:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f740c00f070 con 0x7f74240fc0e0 2021-11-20T07:33:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.389+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f740c025ac0 con 0x7f74240fc0e0 2021-11-20T07:33:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.390+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424045ad0 con 0x7f74240fc0e0 2021-11-20T07:33:30.398 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.393+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f740c016070 con 0x7f74240fc0e0 2021-11-20T07:33:30.467 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.461+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f74240586e0 con 0x7f74240fc0e0 2021-11-20T07:33:30.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.461+0000 7f741bfff700 1 -- 172.21.15.71:0/1383834089 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f740c025d50 con 0x7f74240fc0e0 2021-11-20T07:33:30.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.468 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:33:30.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsid": "16910ea2-49d4-11ec-8c2c-001a4aab830c", 2021-11-20T07:33:30.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "health": { 2021-11-20T07:33:30.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T07:33:30.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T07:33:30.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T07:33:30.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T07:33:30.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T07:33:30.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 0 2021-11-20T07:33:30.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:30.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T07:33:30.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "smithi071" 2021-11-20T07:33:30.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:30.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-11-20T07:33:30.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T07:33:30.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:30.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T07:33:30.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T07:33:30.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T07:33:30.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:30.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T07:33:30.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T07:33:30.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T07:33:30.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T07:33:30.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T07:33:30.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T07:33:30.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T07:33:30.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T07:33:30.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T07:33:30.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T07:33:30.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T07:33:30.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T07:33:30.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T07:33:30.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T07:33:30.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T07:33:30.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T07:33:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T07:33:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T07:33:30.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T07:33:30.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "available": false, 2021-11-20T07:33:30.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T07:33:30.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T07:33:30.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "iostat", 2021-11-20T07:33:30.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "restful" 2021-11-20T07:33:30.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:30.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:30.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T07:33:30.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:30.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modified": "2021-11-20T07:33:21.631103+0000", 2021-11-20T07:33:30.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:30.481 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:30.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T07:33:30.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:33:30.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 msgr2=0x7f742410d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:30.482 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 --2- 172.21.15.71:0/1383834089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f740c00b1d0 tx=0x7f740c006ee0).stop 2021-11-20T07:33:30.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 shutdown_connections 2021-11-20T07:33:30.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 --2- 172.21.15.71:0/1383834089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74240fc0e0 0x7f742410d720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:30.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 >> 172.21.15.71:0/1383834089 conn(0x7f74240f59e0 msgr2=0x7f74240f75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:30.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 shutdown_connections 2021-11-20T07:33:30.483 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:30.463+0000 7f742a1e4700 1 -- 172.21.15.71:0/1383834089 wait complete. 2021-11-20T07:33:30.858 INFO:teuthology.orchestra.run.smithi071.stderr:mgr not available, waiting (2/15)... 2021-11-20T07:33:31.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:31 smithi071 conmon[33687]: audit 2021-11-20T07:33:30.462709+0000 mon.smithi071 (mon.0) 8 2021-11-20T07:33:31.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:31 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1383834089' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T07:33:32.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: cluster 2021-11-20T07:33:31.924047+0000 mon.smithi071 (mon.0) 9 2021-11-20T07:33:32.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: : cluster [INF] Activating manager daemon smithi071.ynqmwn 2021-11-20T07:33:32.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: cluster 2021-11-20T07:33:31.926665+0000 mon.smithi071 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi071.ynqmwn(active, starting, since 0.002711s) 2021-11-20T07:33:32.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11-20T07:33:32.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 2021-11-20T07:33:31.927534+0000 mon.smithi071 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T07:33:32.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:33:32.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 20T07:33:31.927719+0000 mon.smithi071 (mon 2021-11-20T07:33:32.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: .0) 12 : audit [DBG] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T07:33:32.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:33:32.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: T07:33:31.927878+0000 mon.smithi071 (mon. 2021-11-20T07:33:32.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 0) 13 : audit [DBG] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T07:33:32.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:33:32.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: T07:33:31.928050+0000 mon.smithi071 (mon. 2021-11-20T07:33:32.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 0) 14 : audit [DBG] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:33:32.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:33:32.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 20T07:33:31.928198+0000 mon.smithi071 (mon 2021-11-20T07:33:32.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: .0) 15 : audit [DBG] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi071.ynqmwn", "id": "smithi071.ynqmwn"}]: dispatch 2021-11-20T07:33:32.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:33:32.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: T07:33:31.941427+0000 mon.smithi071 (mon 2021-11-20T07:33:32.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: .0) 16 : cluster [INF] Manager daemon smithi071.ynqmwn is now available 2021-11-20T07:33:32.573 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:33:32.573 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 20T07:33:31.949346+0000 mon.smithi071 ( 2021-11-20T07:33:32.573 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:32.573 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:33:32.574 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: -20T07:33:31.953204+0000 mon.smithi071 2021-11-20T07:33:32.574 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:33:32.574 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:33:32.574 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 11-20T07:33:31.954287+0000 2021-11-20T07:33:32.574 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: mon.smithi071 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:33:32.575 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 2021-11-20T07:33:32.575 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021 2021-11-20T07:33:32.575 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: -11-20T07:33:31.958819+0000 mon.smithi071 (mon.0 2021-11-20T07:33:32.575 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: ) 20 : audit [INF] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:32.576 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:33:32.577 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: :33:31.960637+0000 mon.smithi071 (mon.0) 2021-11-20T07:33:32.577 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:32 smithi071 conmon[33687]: 21 : audit [INF] from='mgr.14100 172.21.15.71:0/1325540547' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:32.860 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T07:33:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71523ea700 1 Processor -- start 2021-11-20T07:33:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71523ea700 1 -- start start 2021-11-20T07:33:33.379 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71523ea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71523ea700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0fb810 con 0x7f714c0faf00 2021-11-20T07:33:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71513e8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71513e8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46872/0 (socket says 172.21.15.71:46872) 2021-11-20T07:33:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71513e8700 1 -- 172.21.15.71:0/30383569 learned_addr learned my addr 172.21.15.71:0/30383569 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.373+0000 7f71513e8700 1 -- 172.21.15.71:0/30383569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c0fb950 con 0x7f714c0faf00 2021-11-20T07:33:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71513e8700 1 --2- 172.21.15.71:0/30383569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f714800b5f0 tx=0x7f7148006ee0).ready entity=mon.0 client_cookie=1fd06d65da3b33c5 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f713bfff700 1 -- 172.21.15.71:0/30383569 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714800d600 con 0x7f714c0faf00 2021-11-20T07:33:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f713bfff700 1 -- 172.21.15.71:0/30383569 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f714800dbd0 con 0x7f714c0faf00 2021-11-20T07:33:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f713bfff700 1 -- 172.21.15.71:0/30383569 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148011c00 con 0x7f714c0faf00 2021-11-20T07:33:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71523ea700 1 -- 172.21.15.71:0/30383569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 msgr2=0x7f714c0fb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71523ea700 1 --2- 172.21.15.71:0/30383569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f714800b5f0 tx=0x7f7148006ee0).stop 2021-11-20T07:33:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71523ea700 1 -- 172.21.15.71:0/30383569 shutdown_connections 2021-11-20T07:33:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71523ea700 1 --2- 172.21.15.71:0/30383569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c0fb320 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.374+0000 7f71523ea700 1 -- 172.21.15.71:0/30383569 >> 172.21.15.71:0/30383569 conn(0x7f714c0f5870 msgr2=0x7f714c0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 -- 172.21.15.71:0/30383569 shutdown_connections 2021-11-20T07:33:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 -- 172.21.15.71:0/30383569 wait complete. 2021-11-20T07:33:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 Processor -- start 2021-11-20T07:33:33.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 -- start start 2021-11-20T07:33:33.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:33.386 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71523ea700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c108b20 con 0x7f714c0faf00 2021-11-20T07:33:33.387 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71513e8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:33.387 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71513e8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46874/0 (socket says 172.21.15.71:46874) 2021-11-20T07:33:33.387 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.375+0000 7f71513e8700 1 -- 172.21.15.71:0/1381330469 learned_addr learned my addr 172.21.15.71:0/1381330469 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:33.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f71513e8700 1 -- 172.21.15.71:0/1381330469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714800b200 con 0x7f714c0faf00 2021-11-20T07:33:33.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f71513e8700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7148000c00 tx=0x7f7148006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:33.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148011690 con 0x7f714c0faf00 2021-11-20T07:33:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f714800d9f0 con 0x7f714c0faf00 2021-11-20T07:33:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c108d20 con 0x7f714c0faf00 2021-11-20T07:33:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714801b710 con 0x7f714c0faf00 2021-11-20T07:33:33.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.376+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c109130 con 0x7f714c0faf00 2021-11-20T07:33:33.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f714800f070 con 0x7f714c0faf00 2021-11-20T07:33:33.393 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f7139ffb700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 0x7f71340343b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:33.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7148040fc0 con 0x7f714c0faf00 2021-11-20T07:33:33.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f7150be7700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 0x7f71340343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:33.394 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c059100 con 0x7f714c0faf00 2021-11-20T07:33:33.395 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.377+0000 7f7150be7700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 0x7f71340343b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:33.395 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.381+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7148016070 con 0x7f714c0faf00 2021-11-20T07:33:33.597 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.583+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f714c045ad0 con 0x7f714c0faf00 2021-11-20T07:33:33.598 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.584+0000 7f7139ffb700 1 -- 172.21.15.71:0/1381330469 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f7148022900 con 0x7f714c0faf00 2021-11-20T07:33:33.598 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.598 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:33:33.599 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsid": "16910ea2-49d4-11ec-8c2c-001a4aab830c", 2021-11-20T07:33:33.599 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "health": { 2021-11-20T07:33:33.600 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T07:33:33.600 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T07:33:33.600 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T07:33:33.601 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.601 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T07:33:33.602 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T07:33:33.602 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 0 2021-11-20T07:33:33.602 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:33.603 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T07:33:33.603 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "smithi071" 2021-11-20T07:33:33.604 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:33.604 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "quorum_age": 8, 2021-11-20T07:33:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T07:33:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T07:33:33.607 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T07:33:33.607 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.607 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T07:33:33.608 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:33.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T07:33:33.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T07:33:33.609 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T07:33:33.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T07:33:33.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T07:33:33.610 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T07:33:33.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T07:33:33.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T07:33:33.611 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T07:33:33.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T07:33:33.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T07:33:33.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T07:33:33.612 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T07:33:33.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T07:33:33.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T07:33:33.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T07:33:33.613 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:33.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T07:33:33.614 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T07:33:33.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T07:33:33.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "available": true, 2021-11-20T07:33:33.615 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T07:33:33.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T07:33:33.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "iostat", 2021-11-20T07:33:33.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "restful" 2021-11-20T07:33:33.616 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ], 2021-11-20T07:33:33.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:33.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.617 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T07:33:33.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T07:33:33.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "modified": "2021-11-20T07:33:21.631103+0000", 2021-11-20T07:33:33.618 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "services": {} 2021-11-20T07:33:33.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: }, 2021-11-20T07:33:33.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T07:33:33.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:33:33.619 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 msgr2=0x7f71340343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:33.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 0x7f71340343b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).stop 2021-11-20T07:33:33.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 msgr2=0x7f714c108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:33.620 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7148000c00 tx=0x7f7148006ee0).stop 2021-11-20T07:33:33.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 shutdown_connections 2021-11-20T07:33:33.621 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f7134031ef0 0x7f71340343b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:33.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 --2- 172.21.15.71:0/1381330469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f714c0faf00 0x7f714c108630 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:33.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 >> 172.21.15.71:0/1381330469 conn(0x7f714c0f5870 msgr2=0x7f714c0ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:33.622 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 shutdown_connections 2021-11-20T07:33:33.623 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:33.586+0000 7f71523ea700 1 -- 172.21.15.71:0/1381330469 wait complete. 2021-11-20T07:33:33.899 INFO:teuthology.orchestra.run.smithi071.stderr:mgr is available 2021-11-20T07:33:33.899 INFO:teuthology.orchestra.run.smithi071.stderr:Enabling cephadm module... 2021-11-20T07:33:33.899 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2021-11-20T07:33:34.007 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:33 smithi071 conmon[33687]: cluster 2021-11-20T07:33:32.928803+0000 mon.smithi071 ( 2021-11-20T07:33:34.007 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:33 smithi071 conmon[33687]: mon.0) 22 : cluster [DBG] mgrmap e3: smithi071.ynqmwn(active, since 1.00485s) 2021-11-20T07:33:34.008 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:33 smithi071 conmon[33687]: audit 2021-11-20T07:33:33.585336+0000 mon.smithi071 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.71:0/1381330469' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T07:33:34.332 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.326+0000 7f81801d7700 1 Processor -- start 2021-11-20T07:33:34.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f81801d7700 1 -- start start 2021-11-20T07:33:34.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f81801d7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:34.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f81801d7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81780fcb20 con 0x7f81780fc210 2021-11-20T07:33:34.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f817df73700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:34.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f817df73700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46878/0 (socket says 172.21.15.71:46878) 2021-11-20T07:33:34.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f817df73700 1 -- 172.21.15.71:0/2834115909 learned_addr learned my addr 172.21.15.71:0/2834115909 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:34.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f817df73700 1 -- 172.21.15.71:0/2834115909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81780fcc60 con 0x7f81780fc210 2021-11-20T07:33:34.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.327+0000 7f817df73700 1 --2- 172.21.15.71:0/2834115909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f817400b5f0 tx=0x7f8174006ee0).ready entity=mon.0 client_cookie=eeffd601807a6839 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:34.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f817cf71700 1 -- 172.21.15.71:0/2834115909 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817400d600 con 0x7f81780fc210 2021-11-20T07:33:34.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f817cf71700 1 -- 172.21.15.71:0/2834115909 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f817400dbd0 con 0x7f81780fc210 2021-11-20T07:33:34.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f817cf71700 1 -- 172.21.15.71:0/2834115909 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8174011c00 con 0x7f81780fc210 2021-11-20T07:33:34.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 -- 172.21.15.71:0/2834115909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 msgr2=0x7f81780fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:34.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 --2- 172.21.15.71:0/2834115909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f817400b5f0 tx=0x7f8174006ee0).stop 2021-11-20T07:33:34.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 -- 172.21.15.71:0/2834115909 shutdown_connections 2021-11-20T07:33:34.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 --2- 172.21.15.71:0/2834115909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f81780fc630 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:34.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 -- 172.21.15.71:0/2834115909 >> 172.21.15.71:0/2834115909 conn(0x7f81780f5a80 msgr2=0x7f81780f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:34.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 -- 172.21.15.71:0/2834115909 shutdown_connections 2021-11-20T07:33:34.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.328+0000 7f81801d7700 1 -- 172.21.15.71:0/2834115909 wait complete. 2021-11-20T07:33:34.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f81801d7700 1 Processor -- start 2021-11-20T07:33:34.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f81801d7700 1 -- start start 2021-11-20T07:33:34.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f81801d7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:34.338 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f81801d7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817810d0c0 con 0x7f81780fc210 2021-11-20T07:33:34.338 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f817df73700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:34.338 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f817df73700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46880/0 (socket says 172.21.15.71:46880) 2021-11-20T07:33:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f817df73700 1 -- 172.21.15.71:0/3249511896 learned_addr learned my addr 172.21.15.71:0/3249511896 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.329+0000 7f817df73700 1 -- 172.21.15.71:0/3249511896 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817400b200 con 0x7f81780fc210 2021-11-20T07:33:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f817df73700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f817400e040 tx=0x7f8174006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817401b410 con 0x7f81780fc210 2021-11-20T07:33:34.339 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f817401b9e0 con 0x7f81780fc210 2021-11-20T07:33:34.340 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817401a770 con 0x7f81780fc210 2021-11-20T07:33:34.340 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817810d2c0 con 0x7f81780fc210 2021-11-20T07:33:34.340 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.330+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817810d6d0 con 0x7f81780fc210 2021-11-20T07:33:34.341 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f817400f070 con 0x7f81780fc210 2021-11-20T07:33:34.341 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f81667fc700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 0x7f8160032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:34.341 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8178045ad0 con 0x7f81780fc210 2021-11-20T07:33:34.341 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8174040e10 con 0x7f81780fc210 2021-11-20T07:33:34.342 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f817d772700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 0x7f8160032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:34.342 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.331+0000 7f817d772700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 0x7f8160032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f81680096a0 tx=0x7f8168006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:34.343 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.335+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8174019d20 con 0x7f81780fc210 2021-11-20T07:33:34.522 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.519+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8178101d70 con 0x7f81780fc210 2021-11-20T07:33:34.930 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.929+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f817401bb50 con 0x7f81780fc210 2021-11-20T07:33:34.931 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.930+0000 7f81667fc700 1 -- 172.21.15.71:0/3249511896 <== mon.0 v2:172.21.15.71:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8174023ac0 con 0x7f81780fc210 2021-11-20T07:33:34.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.931+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 msgr2=0x7f8160032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:34.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.931+0000 7f81801d7700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 0x7f8160032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f81680096a0 tx=0x7f8168006b40).stop 2021-11-20T07:33:34.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.931+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 msgr2=0x7f817810cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:34.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.931+0000 7f81801d7700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f817400e040 tx=0x7f8174006ee0).stop 2021-11-20T07:33:34.935 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 shutdown_connections 2021-11-20T07:33:34.935 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f816002fca0 0x7f8160032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:34.935 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 --2- 172.21.15.71:0/3249511896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81780fc210 0x7f817810cbd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:34.935 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 >> 172.21.15.71:0/3249511896 conn(0x7f81780f5a80 msgr2=0x7f81780f7550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:34.936 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 shutdown_connections 2021-11-20T07:33:34.936 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:34.932+0000 7f81801d7700 1 -- 172.21.15.71:0/3249511896 wait complete. 2021-11-20T07:33:35.231 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T07:33:35.244 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:34 smithi071 conmon[33687]: audit 2021-11-20T07:33:34.521052+0000 mon.smithi071 (mon.0 2021-11-20T07:33:35.244 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:34 smithi071 conmon[33687]: ) 24 : audit [INF] from='client.? 172.21.15.71:0/3249511896' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-11-20T07:33:35.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1916efa700 1 Processor -- start 2021-11-20T07:33:35.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1916efa700 1 -- start start 2021-11-20T07:33:35.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1916efa700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:35.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1916efa700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19100fc6c0 con 0x7f191006fc30 2021-11-20T07:33:35.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1914c96700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:35.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1914c96700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46894/0 (socket says 172.21.15.71:46894) 2021-11-20T07:33:35.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.668+0000 7f1914c96700 1 -- 172.21.15.71:0/2831431640 learned_addr learned my addr 172.21.15.71:0/2831431640 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:35.673 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.669+0000 7f1914c96700 1 -- 172.21.15.71:0/2831431640 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910101130 con 0x7f191006fc30 2021-11-20T07:33:35.673 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.669+0000 7f1914c96700 1 --2- 172.21.15.71:0/2831431640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f190400b5f0 tx=0x7f1904006ee0).ready entity=mon.0 client_cookie=92c94923ee9ef284 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:35.673 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.669+0000 7f190f7fe700 1 -- 172.21.15.71:0/2831431640 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f190400d600 con 0x7f191006fc30 2021-11-20T07:33:35.673 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.669+0000 7f190f7fe700 1 -- 172.21.15.71:0/2831431640 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f190400dbd0 con 0x7f191006fc30 2021-11-20T07:33:35.674 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.669+0000 7f190f7fe700 1 -- 172.21.15.71:0/2831431640 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1904011c00 con 0x7f191006fc30 2021-11-20T07:33:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 -- 172.21.15.71:0/2831431640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 msgr2=0x7f1910100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 --2- 172.21.15.71:0/2831431640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f190400b5f0 tx=0x7f1904006ee0).stop 2021-11-20T07:33:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 -- 172.21.15.71:0/2831431640 shutdown_connections 2021-11-20T07:33:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 --2- 172.21.15.71:0/2831431640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f1910100c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 -- 172.21.15.71:0/2831431640 >> 172.21.15.71:0/2831431640 conn(0x7f19100f6280 msgr2=0x7f19100f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:35.678 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 -- 172.21.15.71:0/2831431640 shutdown_connections 2021-11-20T07:33:35.678 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.670+0000 7f1916efa700 1 -- 172.21.15.71:0/2831431640 wait complete. 2021-11-20T07:33:35.678 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1916efa700 1 Processor -- start 2021-11-20T07:33:35.678 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1916efa700 1 -- start start 2021-11-20T07:33:35.679 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1916efa700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:35.679 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1916efa700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191010d090 con 0x7f191006fc30 2021-11-20T07:33:35.679 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1914c96700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:35.679 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1914c96700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46896/0 (socket says 172.21.15.71:46896) 2021-11-20T07:33:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.671+0000 7f1914c96700 1 -- 172.21.15.71:0/3420920068 learned_addr learned my addr 172.21.15.71:0/3420920068 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f1914c96700 1 -- 172.21.15.71:0/3420920068 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190400b200 con 0x7f191006fc30 2021-11-20T07:33:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f1914c96700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f190400e040 tx=0x7f1904006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f190401b410 con 0x7f191006fc30 2021-11-20T07:33:35.680 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f190401b9e0 con 0x7f191006fc30 2021-11-20T07:33:35.681 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191010d290 con 0x7f191006fc30 2021-11-20T07:33:35.681 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f190401a770 con 0x7f191006fc30 2021-11-20T07:33:35.681 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.672+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191010d6a0 con 0x7f191006fc30 2021-11-20T07:33:35.681 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f190400f070 con 0x7f191006fc30 2021-11-20T07:33:35.681 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f190d7fa700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 0x7f18f4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:35.682 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1904041dc0 con 0x7f191006fc30 2021-11-20T07:33:35.682 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f190ffff700 1 -- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 msgr2=0x7f18f4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:35.682 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f190ffff700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 0x7f18f4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:33:35.682 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.673+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1910045ad0 con 0x7f191006fc30 2021-11-20T07:33:35.683 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.677+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1904016070 con 0x7f191006fc30 2021-11-20T07:33:35.873 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.870+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1910106ac0 con 0x7f191006fc30 2021-11-20T07:33:35.873 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.871+0000 7f190d7fa700 1 -- 172.21.15.71:0/3420920068 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1904024410 con 0x7f191006fc30 2021-11-20T07:33:35.873 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:33:35.874 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 4, 2021-11-20T07:33:35.874 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "available": true, 2021-11-20T07:33:35.874 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "active_name": "smithi071.ynqmwn", 2021-11-20T07:33:35.874 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T07:33:35.874 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:33:35.876 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.872+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 msgr2=0x7f18f4032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:33:35.876 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.872+0000 7f1916efa700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 0x7f18f4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:33:35.876 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.872+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 msgr2=0x7f191010cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:35.877 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.872+0000 7f1916efa700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f190400e040 tx=0x7f1904006ee0).stop 2021-11-20T07:33:35.877 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 shutdown_connections 2021-11-20T07:33:35.877 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f18f402fd50 0x7f18f4032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:33:35.877 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 --2- 172.21.15.71:0/3420920068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f191006fc30 0x7f191010cba0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:35.878 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 >> 172.21.15.71:0/3420920068 conn(0x7f19100f6280 msgr2=0x7f19100f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:35.878 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 shutdown_connections 2021-11-20T07:33:35.878 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:35.873+0000 7f1916efa700 1 -- 172.21.15.71:0/3420920068 wait complete. 2021-11-20T07:33:36.234 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:35 smithi071 conmon[33687]: audit 2021-11-20T07:33:34.931253+0000 mon.smithi071 (mon.0) 2021-11-20T07:33:36.234 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:35 smithi071 conmon[33687]: 25 : audit [INF] from='client.? 172.21.15.71:0/3249511896' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-11-20T07:33:36.235 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:35 smithi071 conmon[33687]: cluster 2021-11-20T07:33:34.931312+0000 mon.smithi071 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi071.ynqmwn(active, since 3s) 2021-11-20T07:33:36.235 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:35 smithi071 conmon[33687]: audit 2021-11-20T07:33:35.872079+0000 mon.smithi071 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.71:0/3420920068' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T07:33:36.268 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for the mgr to restart... 2021-11-20T07:33:36.269 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mgr epoch 4... 2021-11-20T07:33:36.269 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T07:33:36.763 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.760+0000 7f6d8d2ad700 1 Processor -- start 2021-11-20T07:33:36.764 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.760+0000 7f6d8d2ad700 1 -- start start 2021-11-20T07:33:36.764 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d8d2ad700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:36.764 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d8d2ad700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d880fdbb0 con 0x7f6d880fd2a0 2021-11-20T07:33:36.764 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d86ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:36.765 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d86ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46902/0 (socket says 172.21.15.71:46902) 2021-11-20T07:33:36.765 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d86ffd700 1 -- 172.21.15.71:0/1045369460 learned_addr learned my addr 172.21.15.71:0/1045369460 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:36.765 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d86ffd700 1 -- 172.21.15.71:0/1045369460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880fdcf0 con 0x7f6d880fd2a0 2021-11-20T07:33:36.765 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.761+0000 7f6d86ffd700 1 --2- 172.21.15.71:0/1045369460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6d7800b5f0 tx=0x7f6d78006ee0).ready entity=mon.0 client_cookie=2d3ae39b357ecd7f server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:36.766 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d85ffb700 1 -- 172.21.15.71:0/1045369460 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7800d600 con 0x7f6d880fd2a0 2021-11-20T07:33:36.766 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d85ffb700 1 -- 172.21.15.71:0/1045369460 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6d7800dbd0 con 0x7f6d880fd2a0 2021-11-20T07:33:36.766 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d85ffb700 1 -- 172.21.15.71:0/1045369460 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d78011c00 con 0x7f6d880fd2a0 2021-11-20T07:33:36.770 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1045369460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 msgr2=0x7f6d880fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:36.770 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1045369460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6d7800b5f0 tx=0x7f6d78006ee0).stop 2021-11-20T07:33:36.771 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1045369460 shutdown_connections 2021-11-20T07:33:36.771 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1045369460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d880fd6c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:36.771 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.762+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1045369460 >> 172.21.15.71:0/1045369460 conn(0x7f6d880f5a80 msgr2=0x7f6d880f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:36.772 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1045369460 shutdown_connections 2021-11-20T07:33:36.772 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1045369460 wait complete. 2021-11-20T07:33:36.772 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 Processor -- start 2021-11-20T07:33:36.773 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 -- start start 2021-11-20T07:33:36.773 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:36.774 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.763+0000 7f6d8d2ad700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8810aef0 con 0x7f6d880fd2a0 2021-11-20T07:33:36.774 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d86ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:36.775 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d86ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46904/0 (socket says 172.21.15.71:46904) 2021-11-20T07:33:36.775 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d86ffd700 1 -- 172.21.15.71:0/1746502738 learned_addr learned my addr 172.21.15.71:0/1746502738 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:36.776 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d86ffd700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7800b200 con 0x7f6d880fd2a0 2021-11-20T07:33:36.776 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d86ffd700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6d7800b1d0 tx=0x7f6d78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:36.776 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d78011690 con 0x7f6d880fd2a0 2021-11-20T07:33:36.777 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8810b0f0 con 0x7f6d880fd2a0 2021-11-20T07:33:36.777 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8810b500 con 0x7f6d880fd2a0 2021-11-20T07:33:36.778 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6d7801b880 con 0x7f6d880fd2a0 2021-11-20T07:33:36.778 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.764+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7801a650 con 0x7f6d880fd2a0 2021-11-20T07:33:36.779 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6d7800f070 con 0x7f6d880fd2a0 2021-11-20T07:33:36.779 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d7f7fe700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d680328d0 con 0x7f6d6802fd50 2021-11-20T07:33:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6d78041010 con 0x7f6d880fd2a0 2021-11-20T07:33:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d867fc700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.765+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:33:36.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.966+0000 7f6d867fc700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:36.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:36.966+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T07:33:37.368 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:37.367+0000 7f6d867fc700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:37.369 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:37.367+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T07:33:38.169 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:38.168+0000 7f6d867fc700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:38.170 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:38.168+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T07:33:39.771 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:39.769+0000 7f6d867fc700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/896254714 2021-11-20T07:33:39.772 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:39.769+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T07:33:39.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:39.908+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f6d7801b9f0 con 0x7f6d880fd2a0 2021-11-20T07:33:39.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:39.908+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 msgr2=0x7f6d68032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:33:39.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:39.908+0000 7f6d7f7fe700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:33:40.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: cluster 2021-11-20T07:33:39.857124+0000 mon.smithi071 ( 2021-11-20T07:33:40.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: mon.0) 28 : cluster [INF] Active manager daemon smithi071.ynqmwn restarted 2021-11-20T07:33:40.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: cluster 2021-11-20T07:33:39.857546+0000 mon.smithi071 (mon.0) 29 : cluster [INF] 2021-11-20T07:33:40.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: Activating manager daemon smithi071.ynqmwn 2021-11-20T07:33:40.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:33:40.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: T07:33:39.859390+0000 mon.smithi071 (mon.0) 30 : 2021-11-20T07:33:40.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:39 smithi071 conmon[33687]: cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-11-20T07:33:40.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.918+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f6d78042cc0 con 0x7f6d880fd2a0 2021-11-20T07:33:40.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.918+0000 7f6d7f7fe700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 0x7f6d68035c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:40.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.918+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d680328d0 con 0x7f6d68033870 2021-11-20T07:33:40.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.919+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 0x7f6d68035c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:40.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.919+0000 7f6d867fc700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 0x7f6d68035c60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6d70006fd0 tx=0x7f6d70006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:40.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.920+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f6d680328d0 con 0x7f6d68033870 2021-11-20T07:33:40.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.926+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6d88059100 con 0x7f6d68033870 2021-11-20T07:33:40.930 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.926+0000 7f6d7f7fe700 1 -- 172.21.15.71:0/1746502738 <== mgr.14114 v2:172.21.15.71:6800/2583959496 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6d88059100 con 0x7f6d68033870 2021-11-20T07:33:40.931 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:33:40.931 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2021-11-20T07:33:40.931 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T07:33:40.931 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:33:40.932 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 msgr2=0x7f6d68035c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:40.932 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 0x7f6d68035c60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6d70006fd0 tx=0x7f6d70006cf0).stop 2021-11-20T07:33:40.932 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 msgr2=0x7f6d8810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:40.933 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6d7800b1d0 tx=0x7f6d78006ee0).stop 2021-11-20T07:33:40.933 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 shutdown_connections 2021-11-20T07:33:40.933 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f6d68033870 0x7f6d68035c60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:40.933 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d880fd2a0 0x7f6d8810aa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:40.933 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 --2- 172.21.15.71:0/1746502738 >> [v2:172.21.15.71:6800/896254714,v1:172.21.15.71:6801/896254714] conn(0x7f6d6802fd50 0x7f6d68032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:33:40.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 >> 172.21.15.71:0/1746502738 conn(0x7f6d880f5a80 msgr2=0x7f6d880ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:40.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 shutdown_connections 2021-11-20T07:33:40.934 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:40.927+0000 7f6d8d2ad700 1 -- 172.21.15.71:0/1746502738 wait complete. 2021-11-20T07:33:41.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: cluster 2021-11-20T07:33:39.910530+0000 mon.smithi071 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi071.ynqmwn(active, starting, since 0.0530507s) 2021-11-20T07:33:41.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07:33:39.912219+0000 mon.smithi071 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:33:41.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07:33:39.912737+0000 mon.smithi071 (mon.0) 33 : audit [DBG] 2021-11-20T07:33:41.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi071.ynqmwn", "id": "smithi071.ynqmwn"}]: dispatch 2021-11-20T07:33:41.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:33:41.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: T07:33:39.913419+0000 mon.smithi071 (mon.0) 2021-11-20T07:33:41.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 34 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T07:33:41.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:33:41.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 11-20T07:33:39.913595 2021-11-20T07:33:41.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T07:33:41.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 2021-11-20T07:33:41.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:33:41.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 33:39.913750+0000 mon.smithi071 (mon.0) 2021-11-20T07:33:41.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 36 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T07:33:41.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:33:41.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: -20T07:33:39.927596+0000 2021-11-20T07:33:41.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: mon.smithi071 (mon.0) 37 : cluster [INF] 2021-11-20T07:33:41.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: Manager daemon smithi071.ynqmwn is now available 2021-11-20T07:33:41.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:33:41.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 33:39.943378+0000 mon.smithi071 (mon. 2021-11-20T07:33:41.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 0) 38 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:41.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021 2021-11-20T07:33:41.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: -11-20T07:33:39 2021-11-20T07:33:41.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: .948942+0000 mon.smithi071 (mon.0) 39 : audit 2021-11-20T07:33:41.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:41.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:33:41.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 07:33:39.954363+0000 mon.smithi071 ( 2021-11-20T07:33:41.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:41.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07:33:41.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 2021-11-20T07:33: 2021-11-20T07:33:41.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 40.044681+0000 mon.smithi071 (mon.0) 2021-11-20T07:33:41.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 41 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:41.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07:33:41.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 2021-11-20T07:33: 2021-11-20T07:33:41.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 40.045359+0000 mon.smithi071 (mon.0 2021-11-20T07:33:41.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: ) 42 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:33:41.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T07:33:41.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 2021-11-20T07:33: 2021-11-20T07:33:41.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 40.046819+0000 mon.smithi071 (mon.0) 43 2021-11-20T07:33:41.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:33:41.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021 2021-11-20T07:33:41.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: -11-20T07:33:40 2021-11-20T07:33:41.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: .048383+0000 mon.smithi071 (mon.0) 44 : audit 2021-11-20T07:33:41.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:33:41.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:33:41.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 07:33:40.058069+0000 2021-11-20T07:33:41.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: mon.smithi071 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:33:41.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: 2021-11-20T07:33:41.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:33:41.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: T07:33:40.059211+0000 mon.smithi071 2021-11-20T07:33:41.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:40 smithi071 conmon[33687]: (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:33:42.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:41 smithi071 conmon[33687]: audit 2021-11-20T07:33:40.921101+0000 mgr.smithi071.ynqmwn (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T07:33:42.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:41 smithi071 conmon[33687]: cluster 2021-11-20T07:33:40.921936+0000 mon.smithi071 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi071.ynqmwn(active, since 1.06445s) 2021-11-20T07:33:42.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:41 smithi071 conmon[33687]: audit 2021-11-20T07:33:40.927888+0000 mgr.smithi071.ynqmwn 2021-11-20T07:33:42.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:41 smithi071 conmon[33687]: (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T07:33:42.816 INFO:teuthology.orchestra.run.smithi071.stderr:mgr epoch 4 is available 2021-11-20T07:33:42.816 INFO:teuthology.orchestra.run.smithi071.stderr:Setting orchestrator backend to cephadm... 2021-11-20T07:33:42.816 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2021-11-20T07:33:43.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:42 smithi071 conmon[33687]: cluster 2021-11-20T07:33:41.926455+0000 mon.smithi071 (mon.0) 48 : cluster 2021-11-20T07:33:43.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:42 smithi071 conmon[33687]: [DBG] mgrmap e7: smithi071.ynqmwn(active, since 2s) 2021-11-20T07:33:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f5303a06700 1 Processor -- start 2021-11-20T07:33:44.799 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f5303a06700 1 -- start start 2021-11-20T07:33:44.799 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f5303a06700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:44.799 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f5303a06700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0720f0 con 0x7f52fc071cd0 2021-11-20T07:33:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f53017a2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f53017a2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46940/0 (socket says 172.21.15.71:46940) 2021-11-20T07:33:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f53017a2700 1 -- 172.21.15.71:0/2191252306 learned_addr learned my addr 172.21.15.71:0/2191252306 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f53017a2700 1 -- 172.21.15.71:0/2191252306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc070760 con 0x7f52fc071cd0 2021-11-20T07:33:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.548+0000 7f53017a2700 1 --2- 172.21.15.71:0/2191252306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f52f800b5f0 tx=0x7f52f8006ee0).ready entity=mon.0 client_cookie=cf9ef5bddbb6a861 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f52ebfff700 1 -- 172.21.15.71:0/2191252306 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f800d600 con 0x7f52fc071cd0 2021-11-20T07:33:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f52ebfff700 1 -- 172.21.15.71:0/2191252306 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f52f800dbd0 con 0x7f52fc071cd0 2021-11-20T07:33:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f52ebfff700 1 -- 172.21.15.71:0/2191252306 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f8011c00 con 0x7f52fc071cd0 2021-11-20T07:33:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 -- 172.21.15.71:0/2191252306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 msgr2=0x7f52fc070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:44.802 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 --2- 172.21.15.71:0/2191252306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f52f800b5f0 tx=0x7f52f8006ee0).stop 2021-11-20T07:33:44.802 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 -- 172.21.15.71:0/2191252306 shutdown_connections 2021-11-20T07:33:44.802 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 --2- 172.21.15.71:0/2191252306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc070270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:44.802 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 -- 172.21.15.71:0/2191252306 >> 172.21.15.71:0/2191252306 conn(0x7f52fc0f73e0 msgr2=0x7f52fc0f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:44.803 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 -- 172.21.15.71:0/2191252306 shutdown_connections 2021-11-20T07:33:44.803 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.549+0000 7f5303a06700 1 -- 172.21.15.71:0/2191252306 wait complete. 2021-11-20T07:33:44.803 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f5303a06700 1 Processor -- start 2021-11-20T07:33:44.803 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f5303a06700 1 -- start start 2021-11-20T07:33:44.804 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f5303a06700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:44.804 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f5303a06700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc10d0e0 con 0x7f52fc071cd0 2021-11-20T07:33:44.804 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f53017a2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:44.805 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f53017a2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46942/0 (socket says 172.21.15.71:46942) 2021-11-20T07:33:44.805 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f53017a2700 1 -- 172.21.15.71:0/2377833548 learned_addr learned my addr 172.21.15.71:0/2377833548 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:44.805 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f53017a2700 1 -- 172.21.15.71:0/2377833548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f800b200 con 0x7f52fc071cd0 2021-11-20T07:33:44.805 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.550+0000 7f53017a2700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f52f800b1d0 tx=0x7f52f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:44.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.551+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f8011a20 con 0x7f52fc071cd0 2021-11-20T07:33:44.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.551+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f52f801b880 con 0x7f52fc071cd0 2021-11-20T07:33:44.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.551+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f801a700 con 0x7f52fc071cd0 2021-11-20T07:33:44.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.551+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc10d2e0 con 0x7f52fc071cd0 2021-11-20T07:33:44.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.551+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc10d6f0 con 0x7f52fc071cd0 2021-11-20T07:33:44.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.552+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc045ad0 con 0x7f52fc071cd0 2021-11-20T07:33:44.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.552+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f52f800f070 con 0x7f52fc071cd0 2021-11-20T07:33:44.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.552+0000 7f52e9ffb700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 0x7f52e4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:44.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.552+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f52f80418a0 con 0x7f52fc071cd0 2021-11-20T07:33:44.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.552+0000 7f5300fa1700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 0x7f52e4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:44.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.553+0000 7f5300fa1700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 0x7f52e4032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52ec0096a0 tx=0x7f52ec006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:44.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.556+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52f8019720 con 0x7f52fc071cd0 2021-11-20T07:33:44.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.733+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f52fc10dfd0 con 0x7f52e402fdb0 2021-11-20T07:33:44.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.738+0000 7f52e9ffb700 1 -- 172.21.15.71:0/2377833548 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52fc10dfd0 con 0x7f52e402fdb0 2021-11-20T07:33:44.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 msgr2=0x7f52e4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:44.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 0x7f52e4032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52ec0096a0 tx=0x7f52ec006b40).stop 2021-11-20T07:33:44.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 msgr2=0x7f52fc10cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:44.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f52f800b1d0 tx=0x7f52f8006ee0).stop 2021-11-20T07:33:44.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 shutdown_connections 2021-11-20T07:33:44.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f52e402fdb0 0x7f52e4032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:44.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 --2- 172.21.15.71:0/2377833548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52fc071cd0 0x7f52fc10cbf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:44.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 >> 172.21.15.71:0/2377833548 conn(0x7f52fc0f73e0 msgr2=0x7f52fc0f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:44.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 shutdown_connections 2021-11-20T07:33:44.812 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:44.740+0000 7f5303a06700 1 -- 172.21.15.71:0/2377833548 wait complete. 2021-11-20T07:33:46.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:45 smithi071 conmon[33687]: audit 2021-11-20T07:33:44.735662+0000 mgr.smithi071.ynqmwn (mgr.14114) 3 : audit [DBG] 2021-11-20T07:33:46.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:45 smithi071 conmon[33687]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:46.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:45 smithi071 conmon[33687]: audit 2021-11-20T07:33:44.739483+0000 mon.smithi071 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:46.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:45 smithi071 conmon[33687]: audit 2021-11-20T07:33:44.743080+0000 mon.smithi071 (mon 2021-11-20T07:33:46.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:45 smithi071 conmon[33687]: .0) 50 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:33:46.763 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2021-11-20T07:33:48.955 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.900+0000 7f011a772700 1 Processor -- start 2021-11-20T07:33:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.900+0000 7f011a772700 1 -- start start 2021-11-20T07:33:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.900+0000 7f011a772700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.900+0000 7f011a772700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140720f0 con 0x7f0114071cd0 2021-11-20T07:33:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0113fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0113fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46946/0 (socket says 172.21.15.71:46946) 2021-11-20T07:33:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0113fff700 1 -- 172.21.15.71:0/2355347481 learned_addr learned my addr 172.21.15.71:0/2355347481 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0113fff700 1 -- 172.21.15.71:0/2355347481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0114070760 con 0x7f0114071cd0 2021-11-20T07:33:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0113fff700 1 --2- 172.21.15.71:0/2355347481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f010400b5f0 tx=0x7f0104006ee0).ready entity=mon.0 client_cookie=1f2c3ac3a4dbaa49 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:48.958 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0112ffd700 1 -- 172.21.15.71:0/2355347481 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010400d600 con 0x7f0114071cd0 2021-11-20T07:33:48.958 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0112ffd700 1 -- 172.21.15.71:0/2355347481 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f010400dbd0 con 0x7f0114071cd0 2021-11-20T07:33:48.958 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.901+0000 7f0112ffd700 1 -- 172.21.15.71:0/2355347481 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0104011c00 con 0x7f0114071cd0 2021-11-20T07:33:48.958 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 -- 172.21.15.71:0/2355347481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 msgr2=0x7f0114070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:48.958 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 --2- 172.21.15.71:0/2355347481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f010400b5f0 tx=0x7f0104006ee0).stop 2021-11-20T07:33:48.959 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 -- 172.21.15.71:0/2355347481 shutdown_connections 2021-11-20T07:33:48.959 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 --2- 172.21.15.71:0/2355347481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f0114070270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:48.959 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 -- 172.21.15.71:0/2355347481 >> 172.21.15.71:0/2355347481 conn(0x7f01140f73e0 msgr2=0x7f01140f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:48.959 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 -- 172.21.15.71:0/2355347481 shutdown_connections 2021-11-20T07:33:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 -- 172.21.15.71:0/2355347481 wait complete. 2021-11-20T07:33:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.902+0000 7f011a772700 1 Processor -- start 2021-11-20T07:33:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f011a772700 1 -- start start 2021-11-20T07:33:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f011a772700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f011a772700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011410d070 con 0x7f0114071cd0 2021-11-20T07:33:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0113fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0113fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46948/0 (socket says 172.21.15.71:46948) 2021-11-20T07:33:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0113fff700 1 -- 172.21.15.71:0/1612876752 learned_addr learned my addr 172.21.15.71:0/1612876752 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0113fff700 1 -- 172.21.15.71:0/1612876752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010400b200 con 0x7f0114071cd0 2021-11-20T07:33:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0113fff700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f010400e040 tx=0x7f0104006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010400d9f0 con 0x7f0114071cd0 2021-11-20T07:33:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f010401a9f0 con 0x7f0114071cd0 2021-11-20T07:33:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011410d270 con 0x7f0114071cd0 2021-11-20T07:33:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010401b8c0 con 0x7f0114071cd0 2021-11-20T07:33:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.903+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011410d680 con 0x7f0114071cd0 2021-11-20T07:33:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.904+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f010400f070 con 0x7f0114071cd0 2021-11-20T07:33:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.904+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0114045ad0 con 0x7f0114071cd0 2021-11-20T07:33:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.904+0000 7f0110ff9700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 0x7f0100032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.904+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f01040412d0 con 0x7f0114071cd0 2021-11-20T07:33:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.905+0000 7f01137fe700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 0x7f0100032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.905+0000 7f01137fe700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 0x7f0100032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f01080096a0 tx=0x7f0108006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:48.909+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0104022d20 con 0x7f0114071cd0 2021-11-20T07:33:49.061 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.060+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f011410df60 con 0x7f010002fdb0 2021-11-20T07:33:49.062 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.061+0000 7f0110ff9700 1 -- 172.21.15.71:0/1612876752 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f011410df60 con 0x7f010002fdb0 2021-11-20T07:33:49.062 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: value unchanged 2021-11-20T07:33:49.065 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 msgr2=0x7f0100032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:49.066 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 0x7f0100032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f01080096a0 tx=0x7f0108006b40).stop 2021-11-20T07:33:49.066 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 msgr2=0x7f011410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:49.067 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f010400e040 tx=0x7f0104006ee0).stop 2021-11-20T07:33:49.067 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 shutdown_connections 2021-11-20T07:33:49.067 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f010002fdb0 0x7f0100032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:49.068 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 --2- 172.21.15.71:0/1612876752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0114071cd0 0x7f011410cb80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:49.068 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.062+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 >> 172.21.15.71:0/1612876752 conn(0x7f01140f73e0 msgr2=0x7f01140f8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:49.069 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.063+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 shutdown_connections 2021-11-20T07:33:49.069 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.063+0000 7f011a772700 1 -- 172.21.15.71:0/1612876752 wait complete. 2021-11-20T07:33:49.409 INFO:teuthology.orchestra.run.smithi071.stderr:Generating ssh key... 2021-11-20T07:33:49.410 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2021-11-20T07:33:49.831 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f03fff700 1 Processor -- start 2021-11-20T07:33:49.832 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f03fff700 1 -- start start 2021-11-20T07:33:49.832 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f03fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:49.833 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f03fff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040fc7a0 con 0x7f2f040fbe90 2021-11-20T07:33:49.833 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f02ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:49.833 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f02ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46952/0 (socket says 172.21.15.71:46952) 2021-11-20T07:33:49.833 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f02ffd700 1 -- 172.21.15.71:0/2004788672 learned_addr learned my addr 172.21.15.71:0/2004788672 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:49.834 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.827+0000 7f2f02ffd700 1 -- 172.21.15.71:0/2004788672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040fc8e0 con 0x7f2f040fbe90 2021-11-20T07:33:49.834 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f02ffd700 1 --2- 172.21.15.71:0/2004788672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2ef400b5f0 tx=0x7f2ef4006ee0).ready entity=mon.0 client_cookie=478799f190fa6447 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:49.834 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f01ffb700 1 -- 172.21.15.71:0/2004788672 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef400d600 con 0x7f2f040fbe90 2021-11-20T07:33:49.835 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f01ffb700 1 -- 172.21.15.71:0/2004788672 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2ef400dbd0 con 0x7f2f040fbe90 2021-11-20T07:33:49.835 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f01ffb700 1 -- 172.21.15.71:0/2004788672 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef4011c00 con 0x7f2f040fbe90 2021-11-20T07:33:49.836 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f03fff700 1 -- 172.21.15.71:0/2004788672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 msgr2=0x7f2f040fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:49.836 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f03fff700 1 --2- 172.21.15.71:0/2004788672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2ef400b5f0 tx=0x7f2ef4006ee0).stop 2021-11-20T07:33:49.836 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f03fff700 1 -- 172.21.15.71:0/2004788672 shutdown_connections 2021-11-20T07:33:49.836 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f03fff700 1 --2- 172.21.15.71:0/2004788672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f040fc2b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:49.837 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.828+0000 7f2f03fff700 1 -- 172.21.15.71:0/2004788672 >> 172.21.15.71:0/2004788672 conn(0x7f2f040f5850 msgr2=0x7f2f040f7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:49.837 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.829+0000 7f2f03fff700 1 -- 172.21.15.71:0/2004788672 shutdown_connections 2021-11-20T07:33:49.837 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.829+0000 7f2f03fff700 1 -- 172.21.15.71:0/2004788672 wait complete. 2021-11-20T07:33:49.837 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.829+0000 7f2f03fff700 1 Processor -- start 2021-11-20T07:33:49.838 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.829+0000 7f2f03fff700 1 -- start start 2021-11-20T07:33:49.838 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f03fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:49.838 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f03fff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040fa860 con 0x7f2f040fbe90 2021-11-20T07:33:49.838 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f02ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:49.839 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f02ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46954/0 (socket says 172.21.15.71:46954) 2021-11-20T07:33:49.839 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f02ffd700 1 -- 172.21.15.71:0/226415553 learned_addr learned my addr 172.21.15.71:0/226415553 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:49.839 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f02ffd700 1 -- 172.21.15.71:0/226415553 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef400b200 con 0x7f2f040fbe90 2021-11-20T07:33:49.839 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f02ffd700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2ef4000c00 tx=0x7f2ef4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:49.840 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef4011690 con 0x7f2f040fbe90 2021-11-20T07:33:49.840 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f040faa60 con 0x7f2f040fbe90 2021-11-20T07:33:49.842 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.830+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0410dbd0 con 0x7f2f040fbe90 2021-11-20T07:33:49.842 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2ef400d9f0 con 0x7f2f040fbe90 2021-11-20T07:33:49.842 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef401b690 con 0x7f2f040fbe90 2021-11-20T07:33:49.843 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2ef400f070 con 0x7f2f040fbe90 2021-11-20T07:33:49.843 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2eeb7fe700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 0x7f2ef0032220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:49.843 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee4004fa0 con 0x7f2f040fbe90 2021-11-20T07:33:49.843 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2ef4041170 con 0x7f2f040fbe90 2021-11-20T07:33:49.844 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.831+0000 7f2f027fc700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 0x7f2ef0032220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:49.844 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.832+0000 7f2f027fc700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 0x7f2ef0032220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2ef80096a0 tx=0x7f2ef8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:49.844 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.836+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ef4016070 con 0x7f2f040fbe90 2021-11-20T07:33:49.988 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:49.988+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ee40029f0 con 0x7f2ef002fd60 2021-11-20T07:33:50.217 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:50 smithi071 conmon[33687]: audit 2021-11-20T07:33:49.061302+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:33:50.217 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:50 smithi071 conmon[33687]: 14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:50.228 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.227+0000 7f2eeb7fe700 1 -- 172.21.15.71:0/226415553 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ee40029f0 con 0x7f2ef002fd60 2021-11-20T07:33:50.230 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 msgr2=0x7f2ef0032220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:50.230 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 0x7f2ef0032220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2ef80096a0 tx=0x7f2ef8006b40).stop 2021-11-20T07:33:50.231 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 msgr2=0x7f2f0410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:50.231 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2ef4000c00 tx=0x7f2ef4006ee0).stop 2021-11-20T07:33:50.231 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 shutdown_connections 2021-11-20T07:33:50.231 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f2ef002fd60 0x7f2ef0032220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:50.232 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 --2- 172.21.15.71:0/226415553 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2f040fbe90 0x7f2f0410d520 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:50.232 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 >> 172.21.15.71:0/226415553 conn(0x7f2f040f5850 msgr2=0x7f2f040f73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:50.232 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 shutdown_connections 2021-11-20T07:33:50.232 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.229+0000 7f2f03fff700 1 -- 172.21.15.71:0/226415553 wait complete. 2021-11-20T07:33:50.546 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2021-11-20T07:33:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f338c126700 1 Processor -- start 2021-11-20T07:33:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f338c126700 1 -- start start 2021-11-20T07:33:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f338c126700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:50.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f338c126700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840fb3e0 con 0x7f33840faad0 2021-11-20T07:33:50.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f3389ec2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:50.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f3389ec2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46958/0 (socket says 172.21.15.71:46958) 2021-11-20T07:33:50.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.962+0000 7f3389ec2700 1 -- 172.21.15.71:0/2092689828 learned_addr learned my addr 172.21.15.71:0/2092689828 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:50.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f3389ec2700 1 -- 172.21.15.71:0/2092689828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840fb520 con 0x7f33840faad0 2021-11-20T07:33:50.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f3389ec2700 1 --2- 172.21.15.71:0/2092689828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f338000b5f0 tx=0x7f3380006ee0).ready entity=mon.0 client_cookie=541ca3a6f8072026 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:50.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f3388ec0700 1 -- 172.21.15.71:0/2092689828 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f338000d600 con 0x7f33840faad0 2021-11-20T07:33:50.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f3388ec0700 1 -- 172.21.15.71:0/2092689828 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f338000dbd0 con 0x7f33840faad0 2021-11-20T07:33:50.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f3388ec0700 1 -- 172.21.15.71:0/2092689828 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3380011c00 con 0x7f33840faad0 2021-11-20T07:33:50.971 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f338c126700 1 -- 172.21.15.71:0/2092689828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 msgr2=0x7f33840faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:50.971 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f338c126700 1 --2- 172.21.15.71:0/2092689828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f338000b5f0 tx=0x7f3380006ee0).stop 2021-11-20T07:33:50.971 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f338c126700 1 -- 172.21.15.71:0/2092689828 shutdown_connections 2021-11-20T07:33:50.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f338c126700 1 --2- 172.21.15.71:0/2092689828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f33840faef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:50.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.963+0000 7f338c126700 1 -- 172.21.15.71:0/2092689828 >> 172.21.15.71:0/2092689828 conn(0x7f33840f6310 msgr2=0x7f33840f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:50.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 -- 172.21.15.71:0/2092689828 shutdown_connections 2021-11-20T07:33:50.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 -- 172.21.15.71:0/2092689828 wait complete. 2021-11-20T07:33:50.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 Processor -- start 2021-11-20T07:33:50.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 -- start start 2021-11-20T07:33:50.974 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:50.974 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f338c126700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840ff1e0 con 0x7f33840faad0 2021-11-20T07:33:50.975 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f3389ec2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:50.975 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f3389ec2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46960/0 (socket says 172.21.15.71:46960) 2021-11-20T07:33:50.976 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.964+0000 7f3389ec2700 1 -- 172.21.15.71:0/2494805112 learned_addr learned my addr 172.21.15.71:0/2494805112 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:50.976 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f3389ec2700 1 -- 172.21.15.71:0/2494805112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338000b200 con 0x7f33840faad0 2021-11-20T07:33:50.976 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f3389ec2700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f338000e040 tx=0x7f3380006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:50.977 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f338001b410 con 0x7f33840faad0 2021-11-20T07:33:50.977 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f338001b9e0 con 0x7f33840faad0 2021-11-20T07:33:50.977 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840ff380 con 0x7f33840faad0 2021-11-20T07:33:50.978 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f338001a820 con 0x7f33840faad0 2021-11-20T07:33:50.978 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.965+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33840ff790 con 0x7f33840faad0 2021-11-20T07:33:50.978 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f338000f070 con 0x7f33840faad0 2021-11-20T07:33:50.978 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f33727fc700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 0x7f336c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3380041390 con 0x7f33840faad0 2021-11-20T07:33:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3384045ad0 con 0x7f33840faad0 2021-11-20T07:33:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f33896c1700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 0x7f336c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.966+0000 7f33896c1700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 0x7f336c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f33740096a0 tx=0x7f3374006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:50.980 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:50.970+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3380016070 con 0x7f33840faad0 2021-11-20T07:33:51.124 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.123+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3384104920 con 0x7f336c02fdb0 2021-11-20T07:33:51.124 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.124+0000 7f33727fc700 1 -- 172.21.15.71:0/2494805112 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3384104920 con 0x7f336c02fdb0 2021-11-20T07:33:51.125 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMgwV8eM3+Exm91hbL/V6gPvDHkDXVH3CSENx+9UHu8GzR8q+A3TXqVyWUbkd0Odc6lWcTXtZQ25hrEnDxk2S697H7R7iAjjnNQgDo5IsKt6GDx+1UllY69niTytQK6GfKBrlKnAYH8PKYdUtKwT5qP7T4by4le03YcRmqaIGiDmP1pb9F7clXAucCsafpvgDaZWYy2t101w27ryhFsUGSZvthhziE4/pLlJKEDc41z6zMpDuv2d9UIOrcTD4ykS0c8Ceh3xlrRGJfM6yxiwH+b0kOsBu3fsjQ6mwlJheCJb1Cfyim0z0FCV8prb+cOjOtRLZ0yBJTegxcE24i9EwBJMYh/aKeio1veYK4GhL6FvxgaEiEUSLq0brpKSQ7X0a+1DmUchYZafaLdaV5pr1FYndEVkMNEgJ1zD8Li/BKNr1QPBYgzSZbKdZZSBX7boDlvnFQgAk8aP2Hi7DvwsTZsvG/bUmwsglTZsCRMVQt+pSprOUfDYXgaEUg5YIoPP8= ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:33:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 msgr2=0x7f336c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 0x7f336c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f33740096a0 tx=0x7f3374006b40).stop 2021-11-20T07:33:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 msgr2=0x7f3384100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f338000e040 tx=0x7f3380006ee0).stop 2021-11-20T07:33:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 shutdown_connections 2021-11-20T07:33:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f336c02fdb0 0x7f336c032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 --2- 172.21.15.71:0/2494805112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33840faad0 0x7f3384100d60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 >> 172.21.15.71:0/2494805112 conn(0x7f33840f6310 msgr2=0x7f338406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:51.129 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.125+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 shutdown_connections 2021-11-20T07:33:51.129 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.126+0000 7f338c126700 1 -- 172.21.15.71:0/2494805112 wait complete. 2021-11-20T07:33:51.481 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: cephadm 2021-11-20T07:33:49.988808+0000 mgr.smithi071.ynqmwn 2021-11-20T07:33:51.481 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2021-11-20T07:33:51.482 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: audit 2021-11-20T07:33:50.221702+0000 mon.smithi071 (mon.0) 51 : audit 2021-11-20T07:33:51.482 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:51.482 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:33:51.482 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: -20T07:33:50.226999+0000 mon.smithi071 (mon.0) 52 : audit 2021-11-20T07:33:51.483 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:51 smithi071 conmon[33687]: [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:51.483 INFO:teuthology.orchestra.run.smithi071.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-11-20T07:33:51.484 INFO:teuthology.orchestra.run.smithi071.stderr:Adding key to root@localhost authorized_keys... 2021-11-20T07:33:51.484 INFO:teuthology.orchestra.run.smithi071.stderr:Adding host smithi071... 2021-11-20T07:33:51.485 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi071 172.21.15.71 2021-11-20T07:33:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.915+0000 7f29960d0700 1 Processor -- start 2021-11-20T07:33:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.915+0000 7f29960d0700 1 -- start start 2021-11-20T07:33:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.915+0000 7f29960d0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.915+0000 7f29960d0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29900fc9d0 con 0x7f29900fc0c0 2021-11-20T07:33:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.915+0000 7f298f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46964/0 (socket says 172.21.15.71:46964) 2021-11-20T07:33:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298f7fe700 1 -- 172.21.15.71:0/2143981450 learned_addr learned my addr 172.21.15.71:0/2143981450 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298f7fe700 1 -- 172.21.15.71:0/2143981450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29900fcb10 con 0x7f29900fc0c0 2021-11-20T07:33:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298f7fe700 1 --2- 172.21.15.71:0/2143981450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f298000b5f0 tx=0x7f2980006ee0).ready entity=mon.0 client_cookie=d77a9b4606e921c0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298e7fc700 1 -- 172.21.15.71:0/2143981450 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298000d600 con 0x7f29900fc0c0 2021-11-20T07:33:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298e7fc700 1 -- 172.21.15.71:0/2143981450 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f298000dbd0 con 0x7f29900fc0c0 2021-11-20T07:33:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.916+0000 7f298e7fc700 1 -- 172.21.15.71:0/2143981450 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2980011c00 con 0x7f29900fc0c0 2021-11-20T07:33:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- 172.21.15.71:0/2143981450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 msgr2=0x7f29900fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 --2- 172.21.15.71:0/2143981450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f298000b5f0 tx=0x7f2980006ee0).stop 2021-11-20T07:33:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- 172.21.15.71:0/2143981450 shutdown_connections 2021-11-20T07:33:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 --2- 172.21.15.71:0/2143981450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f29900fc4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- 172.21.15.71:0/2143981450 >> 172.21.15.71:0/2143981450 conn(0x7f29900f5a80 msgr2=0x7f29900f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- 172.21.15.71:0/2143981450 shutdown_connections 2021-11-20T07:33:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- 172.21.15.71:0/2143981450 wait complete. 2021-11-20T07:33:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 Processor -- start 2021-11-20T07:33:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.917+0000 7f29960d0700 1 -- start start 2021-11-20T07:33:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f29960d0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f29960d0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29900faa30 con 0x7f29900fc0c0 2021-11-20T07:33:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f298f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f298f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46966/0 (socket says 172.21.15.71:46966) 2021-11-20T07:33:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f298f7fe700 1 -- 172.21.15.71:0/1661706936 learned_addr learned my addr 172.21.15.71:0/1661706936 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f298f7fe700 1 -- 172.21.15.71:0/1661706936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298000b200 con 0x7f29900fc0c0 2021-11-20T07:33:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f298f7fe700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f298000e040 tx=0x7f2980006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298001b410 con 0x7f29900fc0c0 2021-11-20T07:33:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f298001b9e0 con 0x7f29900fc0c0 2021-11-20T07:33:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.918+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298001a770 con 0x7f29900fc0c0 2021-11-20T07:33:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.919+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29900fac30 con 0x7f29900fc0c0 2021-11-20T07:33:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.919+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299010ddc0 con 0x7f29900fc0c0 2021-11-20T07:33:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.919+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f298000f070 con 0x7f29900fc0c0 2021-11-20T07:33:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.920+0000 7f2977fff700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 0x7f297c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.920+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2990045ad0 con 0x7f29900fc0c0 2021-11-20T07:33:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.920+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2980040fe0 con 0x7f29900fc0c0 2021-11-20T07:33:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.920+0000 7f298effd700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 0x7f297c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.920+0000 7f298effd700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 0x7f297c032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f29840096a0 tx=0x7f2984006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:51.924+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2980019770 con 0x7f29900fc0c0 2021-11-20T07:33:52.095 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.095+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi071", "addr": "172.21.15.71", "target": ["mon-mgr", ""]}) v1 -- 0x7f29900fb980 con 0x7f297c02fdb0 2021-11-20T07:33:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:52 smithi071 conmon[33687]: audit 2021-11-20T07:33:51.124174+0000 mgr.smithi071.ynqmwn (mgr.14114 2021-11-20T07:33:52.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:52 smithi071 conmon[33687]: ) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:52.776 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Added host 'smithi071' 2021-11-20T07:33:52.777 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.774+0000 7f2977fff700 1 -- 172.21.15.71:0/1661706936 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f29900fb980 con 0x7f297c02fdb0 2021-11-20T07:33:52.779 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.776+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 msgr2=0x7f297c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:52.779 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.776+0000 7f29960d0700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 0x7f297c032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f29840096a0 tx=0x7f2984006b40).stop 2021-11-20T07:33:52.779 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.776+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 msgr2=0x7f299010d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:52.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.776+0000 7f29960d0700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f298000e040 tx=0x7f2980006ee0).stop 2021-11-20T07:33:52.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 shutdown_connections 2021-11-20T07:33:52.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f297c02fdb0 0x7f297c032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:52.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 --2- 172.21.15.71:0/1661706936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29900fc0c0 0x7f299010d710 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:52.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 >> 172.21.15.71:0/1661706936 conn(0x7f29900f5a80 msgr2=0x7f29900fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:52.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 shutdown_connections 2021-11-20T07:33:52.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:52.777+0000 7f29960d0700 1 -- 172.21.15.71:0/1661706936 wait complete. 2021-11-20T07:33:53.020 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying mon service with default placement... 2021-11-20T07:33:53.020 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2021-11-20T07:33:53.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.464+0000 7f77755fc700 1 Processor -- start 2021-11-20T07:33:53.469 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.464+0000 7f77755fc700 1 -- start start 2021-11-20T07:33:53.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.464+0000 7f77755fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:53.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.464+0000 7f77755fc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700fa9d0 con 0x7f77700fa0c0 2021-11-20T07:33:53.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:53.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46976/0 (socket says 172.21.15.71:46976) 2021-11-20T07:33:53.470 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776effd700 1 -- 172.21.15.71:0/2852564345 learned_addr learned my addr 172.21.15.71:0/2852564345 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:53.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776effd700 1 -- 172.21.15.71:0/2852564345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77700fab10 con 0x7f77700fa0c0 2021-11-20T07:33:53.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776effd700 1 --2- 172.21.15.71:0/2852564345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f776000ab00 tx=0x7f776000a470).ready entity=mon.0 client_cookie=c251e9979639641b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:53.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776dffb700 1 -- 172.21.15.71:0/2852564345 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f776000cba0 con 0x7f77700fa0c0 2021-11-20T07:33:53.471 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776dffb700 1 -- 172.21.15.71:0/2852564345 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f776000cd00 con 0x7f77700fa0c0 2021-11-20T07:33:53.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.465+0000 7f776dffb700 1 -- 172.21.15.71:0/2852564345 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f776000f5e0 con 0x7f77700fa0c0 2021-11-20T07:33:53.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 -- 172.21.15.71:0/2852564345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 msgr2=0x7f77700fa4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:53.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 --2- 172.21.15.71:0/2852564345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f776000ab00 tx=0x7f776000a470).stop 2021-11-20T07:33:53.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 -- 172.21.15.71:0/2852564345 shutdown_connections 2021-11-20T07:33:53.472 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 --2- 172.21.15.71:0/2852564345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f77700fa4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:53.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 -- 172.21.15.71:0/2852564345 >> 172.21.15.71:0/2852564345 conn(0x7f77700f58c0 msgr2=0x7f77700f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:53.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 -- 172.21.15.71:0/2852564345 shutdown_connections 2021-11-20T07:33:53.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.466+0000 7f77755fc700 1 -- 172.21.15.71:0/2852564345 wait complete. 2021-11-20T07:33:53.473 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77755fc700 1 Processor -- start 2021-11-20T07:33:53.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77755fc700 1 -- start start 2021-11-20T07:33:53.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77755fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:53.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77755fc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7770108260 con 0x7f77700fa0c0 2021-11-20T07:33:53.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f776effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:53.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f776effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46978/0 (socket says 172.21.15.71:46978) 2021-11-20T07:33:53.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f776effd700 1 -- 172.21.15.71:0/3547018265 learned_addr learned my addr 172.21.15.71:0/3547018265 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:53.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f776effd700 1 -- 172.21.15.71:0/3547018265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776000a710 con 0x7f77700fa0c0 2021-11-20T07:33:53.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f776effd700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7760000f80 tx=0x7f776000a470).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:53.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7760010780 con 0x7f77700fa0c0 2021-11-20T07:33:53.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7770108400 con 0x7f77700fa0c0 2021-11-20T07:33:53.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f77600108e0 con 0x7f77700fa0c0 2021-11-20T07:33:53.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.467+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7760019600 con 0x7f77700fa0c0 2021-11-20T07:33:53.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.468+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7770108810 con 0x7f77700fa0c0 2021-11-20T07:33:53.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.468+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f776000e040 con 0x7f77700fa0c0 2021-11-20T07:33:53.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.468+0000 7f77577fe700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 0x7f775c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:53.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.468+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7770045ad0 con 0x7f77700fa0c0 2021-11-20T07:33:53.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.468+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7760041e20 con 0x7f77700fa0c0 2021-11-20T07:33:53.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.469+0000 7f776e7fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 0x7f775c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:53.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.469+0000 7f776e7fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 0x7f775c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:53.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.473+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f776002a760 con 0x7f77700fa0c0 2021-11-20T07:33:53.650 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.649+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f7770102170 con 0x7f775c02fd50 2021-11-20T07:33:53.653 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.653+0000 7f77577fe700 1 -- 172.21.15.71:0/3547018265 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7770102170 con 0x7f775c02fd50 2021-11-20T07:33:53.654 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled mon update... 2021-11-20T07:33:53.656 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 msgr2=0x7f775c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:53.656 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 0x7f775c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).stop 2021-11-20T07:33:53.656 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 msgr2=0x7f7770109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:53.657 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7760000f80 tx=0x7f776000a470).stop 2021-11-20T07:33:53.657 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 shutdown_connections 2021-11-20T07:33:53.657 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f775c02fd50 0x7f775c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:53.657 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 --2- 172.21.15.71:0/3547018265 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77700fa0c0 0x7f7770109de0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:53.657 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 >> 172.21.15.71:0/3547018265 conn(0x7f77700f58c0 msgr2=0x7f77700fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:53.658 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 shutdown_connections 2021-11-20T07:33:53.658 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:53.655+0000 7f77755fc700 1 -- 172.21.15.71:0/3547018265 wait complete. 2021-11-20T07:33:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:53 smithi071 conmon[33687]: audit 2021-11-20T07:33:52.095912+0000 mgr.smithi071.ynqmwn (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "addr": "172.21.15.71", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:53 smithi071 conmon[33687]: audit 2021-11-20T07:33:52.773967+0000 mon.smithi071 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:53 smithi071 conmon[33687]: cephadm 2021-11-20T07:33:52.774276+0000 mgr.smithi071.ynqmwn (mgr.14114) 9 : cephadm [INF] Added host smithi071 2021-11-20T07:33:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:53 smithi071 conmon[33687]: audit 2021-11-20T07:33:52.774518+0000 mon.smithi071 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:33:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:53 smithi071 conmon[33687]: audit 2021-11-20T07:33:53.653289+0000 mon.smithi071 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:54.161 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying mgr service with default placement... 2021-11-20T07:33:54.162 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2021-11-20T07:33:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.574+0000 7f636b59e700 1 Processor -- start 2021-11-20T07:33:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.574+0000 7f636b59e700 1 -- start start 2021-11-20T07:33:54.578 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636b59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:54.578 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636b59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0feb80 con 0x7f636c06fbf0 2021-11-20T07:33:54.578 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:54.578 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46986/0 (socket says 172.21.15.71:46986) 2021-11-20T07:33:54.579 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636a59c700 1 -- 172.21.15.71:0/3102066385 learned_addr learned my addr 172.21.15.71:0/3102066385 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:54.579 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636a59c700 1 -- 172.21.15.71:0/3102066385 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0fecc0 con 0x7f636c06fbf0 2021-11-20T07:33:54.579 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636a59c700 1 --2- 172.21.15.71:0/3102066385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f635c00b5f0 tx=0x7f635c006ee0).ready entity=mon.0 client_cookie=1da5b662de3bc1ae server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:54.579 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.575+0000 7f636959a700 1 -- 172.21.15.71:0/3102066385 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f635c00d600 con 0x7f636c06fbf0 2021-11-20T07:33:54.580 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636959a700 1 -- 172.21.15.71:0/3102066385 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f635c00dbd0 con 0x7f636c06fbf0 2021-11-20T07:33:54.580 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636959a700 1 -- 172.21.15.71:0/3102066385 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f635c011c30 con 0x7f636c06fbf0 2021-11-20T07:33:54.580 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636b59e700 1 -- 172.21.15.71:0/3102066385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 msgr2=0x7f636c0fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:54.580 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636b59e700 1 --2- 172.21.15.71:0/3102066385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f635c00b5f0 tx=0x7f635c006ee0).stop 2021-11-20T07:33:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636b59e700 1 -- 172.21.15.71:0/3102066385 shutdown_connections 2021-11-20T07:33:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636b59e700 1 --2- 172.21.15.71:0/3102066385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c0fe690 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.576+0000 7f636b59e700 1 -- 172.21.15.71:0/3102066385 >> 172.21.15.71:0/3102066385 conn(0x7f636c0f6130 msgr2=0x7f636c0f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 -- 172.21.15.71:0/3102066385 shutdown_connections 2021-11-20T07:33:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 -- 172.21.15.71:0/3102066385 wait complete. 2021-11-20T07:33:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 Processor -- start 2021-11-20T07:33:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 -- start start 2021-11-20T07:33:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.577+0000 7f636b59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c10ac60 con 0x7f636c06fbf0 2021-11-20T07:33:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46988/0 (socket says 172.21.15.71:46988) 2021-11-20T07:33:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636a59c700 1 -- 172.21.15.71:0/4287769898 learned_addr learned my addr 172.21.15.71:0/4287769898 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636a59c700 1 -- 172.21.15.71:0/4287769898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c00b200 con 0x7f636c06fbf0 2021-11-20T07:33:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636a59c700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f635c011850 tx=0x7f635c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f635c01b570 con 0x7f636c06fbf0 2021-11-20T07:33:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f635c01bb40 con 0x7f636c06fbf0 2021-11-20T07:33:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f635c01aa10 con 0x7f636c06fbf0 2021-11-20T07:33:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c10ae60 con 0x7f636c06fbf0 2021-11-20T07:33:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.578+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c10b270 con 0x7f636c06fbf0 2021-11-20T07:33:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.579+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f635c00f070 con 0x7f636c06fbf0 2021-11-20T07:33:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.579+0000 7f6362ffd700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 0x7f634c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:54.588 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.579+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c045ad0 con 0x7f636c06fbf0 2021-11-20T07:33:54.588 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.579+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f635c041080 con 0x7f636c06fbf0 2021-11-20T07:33:54.588 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.579+0000 7f6369d9b700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 0x7f634c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:54.588 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.580+0000 7f6369d9b700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 0x7f634c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f63540096a0 tx=0x7f6354006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:54.589 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.583+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f635c06d2b0 con 0x7f636c06fbf0 2021-11-20T07:33:54.773 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.772+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f636c10bb50 con 0x7f634c02fdb0 2021-11-20T07:33:54.777 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.777+0000 7f6362ffd700 1 -- 172.21.15.71:0/4287769898 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f636c10bb50 con 0x7f634c02fdb0 2021-11-20T07:33:54.778 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-11-20T07:33:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 msgr2=0x7f634c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 0x7f634c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f63540096a0 tx=0x7f6354006b40).stop 2021-11-20T07:33:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 msgr2=0x7f636c10a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:54.780 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f635c011850 tx=0x7f635c006ee0).stop 2021-11-20T07:33:54.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 shutdown_connections 2021-11-20T07:33:54.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f634c02fdb0 0x7f634c032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:54.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 --2- 172.21.15.71:0/4287769898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f636c06fbf0 0x7f636c10a770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:54.781 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 >> 172.21.15.71:0/4287769898 conn(0x7f636c0f6130 msgr2=0x7f636c0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:54.782 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 shutdown_connections 2021-11-20T07:33:54.782 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:54.778+0000 7f636b59e700 1 -- 172.21.15.71:0/4287769898 wait complete. 2021-11-20T07:33:55.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:54 smithi071 conmon[33687]: audit 2021-11-20T07:33:53.650366+0000 2021-11-20T07:33:55.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:54 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:55.001 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:54 smithi071 conmon[33687]: cephadm 2021-11-20T07:33:53.651068+0000 mgr.smithi071.ynqmwn (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-11-20T07:33:55.299 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying crash service with default placement... 2021-11-20T07:33:55.299 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2021-11-20T07:33:55.987 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:55 smithi071 conmon[33687]: audit 2021-11-20T07:33:54.773698+0000 mgr.smithi071.ynqmwn (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:55.987 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:55 smithi071 conmon[33687]: cephadm 2021-11-20T07:33:54.774221+0000 mgr.smithi071.ynqmwn (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-11-20T07:33:55.987 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:55 smithi071 conmon[33687]: audit 2021-11-20T07:33:54.776883+0000 mon.smithi071 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:55.987 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:55 smithi071 conmon[33687]: audit 2021-11-20T07:33:55.556161+0000 mon.smithi071 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:55.991 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f500ccfb700 1 Processor -- start 2021-11-20T07:33:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f500ccfb700 1 -- start start 2021-11-20T07:33:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f500ccfb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f500ccfb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50080feb30 con 0x7f500806fbf0 2021-11-20T07:33:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f50077fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f50077fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46992/0 (socket says 172.21.15.71:46992) 2021-11-20T07:33:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.985+0000 7f50077fe700 1 -- 172.21.15.71:0/270032673 learned_addr learned my addr 172.21.15.71:0/270032673 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.986+0000 7f50077fe700 1 -- 172.21.15.71:0/270032673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50080fec70 con 0x7f500806fbf0 2021-11-20T07:33:55.994 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.986+0000 7f50077fe700 1 --2- 172.21.15.71:0/270032673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4ff800b5f0 tx=0x7f4ff8006ee0).ready entity=mon.0 client_cookie=27317a7d25fc0aa2 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:55.994 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.986+0000 7f50067fc700 1 -- 172.21.15.71:0/270032673 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff800d600 con 0x7f500806fbf0 2021-11-20T07:33:55.994 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.986+0000 7f50067fc700 1 -- 172.21.15.71:0/270032673 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ff800dc10 con 0x7f500806fbf0 2021-11-20T07:33:55.994 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.986+0000 7f50067fc700 1 -- 172.21.15.71:0/270032673 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8011d00 con 0x7f500806fbf0 2021-11-20T07:33:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 -- 172.21.15.71:0/270032673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 msgr2=0x7f50080fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 --2- 172.21.15.71:0/270032673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4ff800b5f0 tx=0x7f4ff8006ee0).stop 2021-11-20T07:33:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 -- 172.21.15.71:0/270032673 shutdown_connections 2021-11-20T07:33:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 --2- 172.21.15.71:0/270032673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fe640 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 -- 172.21.15.71:0/270032673 >> 172.21.15.71:0/270032673 conn(0x7f50080f60e0 msgr2=0x7f50080f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:55.996 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 -- 172.21.15.71:0/270032673 shutdown_connections 2021-11-20T07:33:55.996 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 -- 172.21.15.71:0/270032673 wait complete. 2021-11-20T07:33:55.997 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.987+0000 7f500ccfb700 1 Processor -- start 2021-11-20T07:33:55.997 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f500ccfb700 1 -- start start 2021-11-20T07:33:55.998 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f500ccfb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:55.998 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f500ccfb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50080fa540 con 0x7f500806fbf0 2021-11-20T07:33:55.999 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f50077fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:55.999 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f50077fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46994/0 (socket says 172.21.15.71:46994) 2021-11-20T07:33:55.999 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f50077fe700 1 -- 172.21.15.71:0/230477490 learned_addr learned my addr 172.21.15.71:0/230477490 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f50077fe700 1 -- 172.21.15.71:0/230477490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff800b200 con 0x7f500806fbf0 2021-11-20T07:33:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f50077fe700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4ff800b160 tx=0x7f4ff8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:56.001 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8011e20 con 0x7f500806fbf0 2021-11-20T07:33:56.001 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ff801b8c0 con 0x7f500806fbf0 2021-11-20T07:33:56.002 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.988+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50080fa6e0 con 0x7f500806fbf0 2021-11-20T07:33:56.002 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.989+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff801a790 con 0x7f500806fbf0 2021-11-20T07:33:56.003 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.989+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50080faaf0 con 0x7f500806fbf0 2021-11-20T07:33:56.003 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.989+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4ff800f070 con 0x7f500806fbf0 2021-11-20T07:33:56.003 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.990+0000 7f4feffff700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 0x7f4ff4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:56.004 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.990+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4ff8040f10 con 0x7f500806fbf0 2021-11-20T07:33:56.004 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.990+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5008045ad0 con 0x7f500806fbf0 2021-11-20T07:33:56.005 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.990+0000 7f5006ffd700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 0x7f4ff4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:56.005 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.990+0000 7f5006ffd700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 0x7f4ff4032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ffc0096a0 tx=0x7f4ffc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:56.007 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:55.994+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ff8010750 con 0x7f500806fbf0 2021-11-20T07:33:56.183 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.182+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f50080ff9c0 con 0x7f4ff402fdb0 2021-11-20T07:33:56.188 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.187+0000 7f4feffff700 1 -- 172.21.15.71:0/230477490 <== mgr.14114 v2:172.21.15.71:6800/2583959496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f50080ff9c0 con 0x7f4ff402fdb0 2021-11-20T07:33:56.189 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled crash update... 2021-11-20T07:33:56.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 msgr2=0x7f4ff4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:56.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 0x7f4ff4032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ffc0096a0 tx=0x7f4ffc006b40).stop 2021-11-20T07:33:56.191 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 msgr2=0x7f50080fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:56.192 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4ff800b160 tx=0x7f4ff8006ee0).stop 2021-11-20T07:33:56.192 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 shutdown_connections 2021-11-20T07:33:56.192 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f4ff402fdb0 0x7f4ff4032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:56.192 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 --2- 172.21.15.71:0/230477490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f500806fbf0 0x7f50080fc0c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:56.193 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 >> 172.21.15.71:0/230477490 conn(0x7f50080f60e0 msgr2=0x7f50080f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:56.193 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 shutdown_connections 2021-11-20T07:33:56.193 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:56.189+0000 7f500ccfb700 1 -- 172.21.15.71:0/230477490 wait complete. 2021-11-20T07:33:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:Enabling mgr prometheus module... 2021-11-20T07:33:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2021-11-20T07:33:57.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:56 smithi071 conmon[33687]: audit 2021-11-20T07:33:55.888147+0000 mon.smithi071 ( 2021-11-20T07:33:57.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:56 smithi071 conmon[33687]: mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:57.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:56 smithi071 conmon[33687]: audit 2021-11-20T07:33:56.187546+0000 mon.smithi071 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668cf3b700 1 Processor -- start 2021-11-20T07:33:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668cf3b700 1 -- start start 2021-11-20T07:33:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668cf3b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668cf3b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880fc4d0 con 0x7f66880fbbc0 2021-11-20T07:33:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46998/0 (socket says 172.21.15.71:46998) 2021-11-20T07:33:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.074+0000 7f668659c700 1 -- 172.21.15.71:0/2409985655 learned_addr learned my addr 172.21.15.71:0/2409985655 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.075+0000 7f668659c700 1 -- 172.21.15.71:0/2409985655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880fc610 con 0x7f66880fbbc0 2021-11-20T07:33:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.075+0000 7f668659c700 1 --2- 172.21.15.71:0/2409985655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6678009960 tx=0x7f66780092d0).ready entity=mon.0 client_cookie=a304587d588c3f89 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:57.082 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.075+0000 7f6685d9b700 1 -- 172.21.15.71:0/2409985655 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667800b9c0 con 0x7f66880fbbc0 2021-11-20T07:33:57.083 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.075+0000 7f6685d9b700 1 -- 172.21.15.71:0/2409985655 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6678011400 con 0x7f66880fbbc0 2021-11-20T07:33:57.083 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.075+0000 7f6685d9b700 1 -- 172.21.15.71:0/2409985655 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6678010580 con 0x7f66880fbbc0 2021-11-20T07:33:57.083 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- 172.21.15.71:0/2409985655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 msgr2=0x7f66880fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:57.083 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 --2- 172.21.15.71:0/2409985655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6678009960 tx=0x7f66780092d0).stop 2021-11-20T07:33:57.083 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- 172.21.15.71:0/2409985655 shutdown_connections 2021-11-20T07:33:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 --2- 172.21.15.71:0/2409985655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f66880fbfe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- 172.21.15.71:0/2409985655 >> 172.21.15.71:0/2409985655 conn(0x7f66880f73e0 msgr2=0x7f66880f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- 172.21.15.71:0/2409985655 shutdown_connections 2021-11-20T07:33:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- 172.21.15.71:0/2409985655 wait complete. 2021-11-20T07:33:57.084 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 Processor -- start 2021-11-20T07:33:57.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.076+0000 7f668cf3b700 1 -- start start 2021-11-20T07:33:57.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668cf3b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:57.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668cf3b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668810ae70 con 0x7f66880fbbc0 2021-11-20T07:33:57.085 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:57.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47000/0 (socket says 172.21.15.71:47000) 2021-11-20T07:33:57.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668659c700 1 -- 172.21.15.71:0/1617503334 learned_addr learned my addr 172.21.15.71:0/1617503334 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:57.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668659c700 1 -- 172.21.15.71:0/1617503334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678009570 con 0x7f66880fbbc0 2021-11-20T07:33:57.086 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668659c700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6678011cf0 tx=0x7f66780092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:57.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6678011aa0 con 0x7f66880fbbc0 2021-11-20T07:33:57.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6678010cf0 con 0x7f66880fbbc0 2021-11-20T07:33:57.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f667801bdd0 con 0x7f66880fbbc0 2021-11-20T07:33:57.087 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668810b070 con 0x7f66880fbbc0 2021-11-20T07:33:57.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.077+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668810b480 con 0x7f66880fbbc0 2021-11-20T07:33:57.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.078+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f667800f070 con 0x7f66880fbbc0 2021-11-20T07:33:57.088 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.078+0000 7f667f7fe700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 0x7f6668032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:57.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.078+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6678041a10 con 0x7f66880fbbc0 2021-11-20T07:33:57.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.078+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688045ad0 con 0x7f66880fbbc0 2021-11-20T07:33:57.089 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.078+0000 7f667dd9b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 0x7f6668032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:57.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.079+0000 7f667dd9b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 0x7f6668032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f66700096a0 tx=0x7f6670006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:57.090 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.083+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6678024400 con 0x7f66880fbbc0 2021-11-20T07:33:57.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.280+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f66880fd930 con 0x7f66880fbbc0 2021-11-20T07:33:57.890 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.888+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f667801b440 con 0x7f66880fbbc0 2021-11-20T07:33:57.890 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.889+0000 7f667f7fe700 1 -- 172.21.15.71:0/1617503334 <== mon.0 v2:172.21.15.71:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f6678023b10 con 0x7f66880fbbc0 2021-11-20T07:33:57.894 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.891+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 msgr2=0x7f6668032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:57.894 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.891+0000 7f668cf3b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 0x7f6668032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f66700096a0 tx=0x7f6670006b40).stop 2021-11-20T07:33:57.894 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 msgr2=0x7f668810a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:57.895 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6678011cf0 tx=0x7f66780092d0).stop 2021-11-20T07:33:57.895 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 shutdown_connections 2021-11-20T07:33:57.895 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7f666802fd50 0x7f6668032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:57.895 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 --2- 172.21.15.71:0/1617503334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66880fbbc0 0x7f668810a980 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:57.896 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 >> 172.21.15.71:0/1617503334 conn(0x7f66880f73e0 msgr2=0x7f66880f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:57.896 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 shutdown_connections 2021-11-20T07:33:57.896 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:57.892+0000 7f668cf3b700 1 -- 172.21.15.71:0/1617503334 wait complete. 2021-11-20T07:33:58.059 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: audit 2021-11-20T07:33:56 2021-11-20T07:33:58.059 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: .183812+0000 mgr.smithi071.ynqmwn (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:33:58.060 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: cephadm 2021-11-20T07:33:56.184454+0000 mgr.smithi071.ynqmwn (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2021-11-20T07:33:58.060 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: audit 2021-11-20T07:33:57.281527+0000 mon.smithi071 (mon.0) 60 : 2021-11-20T07:33:58.060 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: audit [INF] from='client.? 172.21.15.71:0/1617503334' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-11-20T07:33:58.060 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: audit 2021-11-20T07:33 2021-11-20T07:33:58.061 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: :57.385853+0000 mon.smithi071 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.71:0/1627956973' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:33:58.061 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:57 smithi071 conmon[33687]: 2021-11-20T07:33:58.373 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying prometheus service with default placement... 2021-11-20T07:33:58.374 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2021-11-20T07:33:58.956 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:58 smithi071 conmon[33687]: audit 2021-11-20T07:33:57.890543+0000 mon.smithi071 (mon.0) 62 : audit [INF] from='client.? 172.21.15.71:0/1617503334' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-11-20T07:33:58.956 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:33:58 smithi071 conmon[33687]: cluster 2021-11-20T07:33:57.890605+0000 mon.smithi071 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi071.ynqmwn(active, since 18s) 2021-11-20T07:33:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.954+0000 7fb09ae2b700 1 Processor -- start 2021-11-20T07:33:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.954+0000 7fb09ae2b700 1 -- start start 2021-11-20T07:33:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.954+0000 7fb09ae2b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.954+0000 7fb09ae2b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940fb480 con 0x7fb0940faae0 2021-11-20T07:33:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb098bc7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb098bc7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47014/0 (socket says 172.21.15.71:47014) 2021-11-20T07:33:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb098bc7700 1 -- 172.21.15.71:0/1009274394 learned_addr learned my addr 172.21.15.71:0/1009274394 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:58.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb098bc7700 1 -- 172.21.15.71:0/1009274394 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0940fb5c0 con 0x7fb0940faae0 2021-11-20T07:33:58.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb098bc7700 1 --2- 172.21.15.71:0/1009274394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb08800b5f0 tx=0x7fb088006ee0).ready entity=mon.0 client_cookie=50f7dc6c3d729684 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:58.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb0937fe700 1 -- 172.21.15.71:0/1009274394 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb08800d600 con 0x7fb0940faae0 2021-11-20T07:33:58.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb0937fe700 1 -- 172.21.15.71:0/1009274394 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08800dc10 con 0x7fb0940faae0 2021-11-20T07:33:58.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.955+0000 7fb0937fe700 1 -- 172.21.15.71:0/1009274394 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb088011cd0 con 0x7fb0940faae0 2021-11-20T07:33:58.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 -- 172.21.15.71:0/1009274394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 msgr2=0x7fb0940faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:33:58.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/1009274394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb08800b5f0 tx=0x7fb088006ee0).stop 2021-11-20T07:33:58.966 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 -- 172.21.15.71:0/1009274394 shutdown_connections 2021-11-20T07:33:58.966 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/1009274394 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb0940faf00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:33:58.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 -- 172.21.15.71:0/1009274394 >> 172.21.15.71:0/1009274394 conn(0x7fb0940f6280 msgr2=0x7fb0940f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:33:58.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 -- 172.21.15.71:0/1009274394 shutdown_connections 2021-11-20T07:33:58.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.956+0000 7fb09ae2b700 1 -- 172.21.15.71:0/1009274394 wait complete. 2021-11-20T07:33:58.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb09ae2b700 1 Processor -- start 2021-11-20T07:33:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb09ae2b700 1 -- start start 2021-11-20T07:33:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb09ae2b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb09ae2b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094109e40 con 0x7fb0940faae0 2021-11-20T07:33:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb098bc7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:33:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb098bc7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47016/0 (socket says 172.21.15.71:47016) 2021-11-20T07:33:58.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb098bc7700 1 -- 172.21.15.71:0/274738276 learned_addr learned my addr 172.21.15.71:0/274738276 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:33:58.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.957+0000 7fb098bc7700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08800b200 con 0x7fb0940faae0 2021-11-20T07:33:58.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb098bc7700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb088006800 tx=0x7fb088006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:33:58.969 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb088011df0 con 0x7fb0940faae0 2021-11-20T07:33:58.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08801b8c0 con 0x7fb0940faae0 2021-11-20T07:33:58.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb08801a800 con 0x7fb0940faae0 2021-11-20T07:33:58.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09410a040 con 0x7fb0940faae0 2021-11-20T07:33:58.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.958+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09410a450 con 0x7fb0940faae0 2021-11-20T07:33:58.970 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb08800f070 con 0x7fb0940faae0 2021-11-20T07:33:58.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb0917fa700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:33:58.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb0880458a0 con 0x7fb0940faae0 2021-11-20T07:33:58.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb093fff700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/2583959496 2021-11-20T07:33:58.972 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:33:58.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.959+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045ad0 con 0x7fb0940faae0 2021-11-20T07:33:58.973 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:58.963+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb088024410 con 0x7fb0940faae0 2021-11-20T07:33:59.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:59.160+0000 7fb093fff700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/2583959496 2021-11-20T07:33:59.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:59.160+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T07:33:59.162 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:59.161+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb094103880 con 0x7fb07802fe50 2021-11-20T07:33:59.563 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:59.561+0000 7fb093fff700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/2583959496 2021-11-20T07:33:59.563 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:33:59.561+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T07:34:00.363 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:00.362+0000 7fb093fff700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/2583959496 2021-11-20T07:34:00.364 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:00.362+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T07:34:01.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:01.963+0000 7fb093fff700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/2583959496 2021-11-20T07:34:01.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:01.963+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T07:34:02.865 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:02.862+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fb088024790 con 0x7fb0940faae0 2021-11-20T07:34:02.865 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:02.863+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 msgr2=0x7fb078032310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:34:02.867 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:02.863+0000 7fb0917fa700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:03.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:02 smithi071 conmon[33687]: cluster 2021-11-20T07:34:02.812108+0000 2021-11-20T07:34:03.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:02 smithi071 conmon[33687]: mon.smithi071 (mon.0) 64 : cluster [INF] Active manager daemon smithi071.ynqmwn restarted 2021-11-20T07:34:03.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:02 smithi071 conmon[33687]: cluster 2021-11-20T07:34:02.812509+0000 mon.smithi071 (mon.0) 65 : cluster [INF] Activating manager daemon smithi071.ynqmwn 2021-11-20T07:34:03.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:02 smithi071 conmon[33687]: cluster 2021-11-20T07:34:02.814144+0000 mon.smithi071 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-11-20T07:34:03.871 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.868+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mon.0 v2:172.21.15.71:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fb088022c10 con 0x7fb0940faae0 2021-11-20T07:34:03.872 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.868+0000 7fb0917fa700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 0x7fb078036060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:03.872 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.869+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 --> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb094103880 con 0x7fb078033c70 2021-11-20T07:34:03.872 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.869+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 0x7fb078036060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:03.873 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.869+0000 7fb093fff700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 0x7fb078036060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb0800036d0 tx=0x7fb080006d60).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:03.875 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.873+0000 7fb0917fa700 1 -- 172.21.15.71:0/274738276 <== mgr.14142 v2:172.21.15.71:6800/4211885430 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fb094103880 con 0x7fb078033c70 2021-11-20T07:34:03.876 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled prometheus update... 2021-11-20T07:34:03.878 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 msgr2=0x7fb078036060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:03.878 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 0x7fb078036060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb0800036d0 tx=0x7fb080006d60).stop 2021-11-20T07:34:03.879 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 msgr2=0x7fb094109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:03.879 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb088006800 tx=0x7fb088006ee0).stop 2021-11-20T07:34:03.879 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 shutdown_connections 2021-11-20T07:34:03.880 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fb078033c70 0x7fb078036060 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:03.880 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0940faae0 0x7fb094109950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:03.880 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 --2- 172.21.15.71:0/274738276 >> [v2:172.21.15.71:6800/2583959496,v1:172.21.15.71:6801/2583959496] conn(0x7fb07802fe50 0x7fb078032310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:03.880 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 >> 172.21.15.71:0/274738276 conn(0x7fb0940f6280 msgr2=0x7fb0940f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:03.881 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 shutdown_connections 2021-11-20T07:34:03.881 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:03.875+0000 7fb09ae2b700 1 -- 172.21.15.71:0/274738276 wait complete. 2021-11-20T07:34:04.195 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying grafana service with default placement... 2021-11-20T07:34:04.196 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2021-11-20T07:34:04.283 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: cluster 2021-11-20T07:34:02.864725+0000 mon.smithi071 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi071.ynqmwn(active, starting, since 0.0522918s) 2021-11-20T07:34:04.283 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:02.867591+0000 mon.smithi071 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:34:04.284 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:02.867773+0000 mon.smithi071 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi071.ynqmwn", "id": "smithi071.ynqmwn"}]: dispatch 2021-11-20T07:34:04.284 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:02.867927+0000 mon.smithi071 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T07:34:04.284 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:02.868047+0000 mon.smithi071 (mon.0) 71 : audit [DBG] 2021-11-20T07:34:04.284 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T07:34:04.284 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:04.285 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 2021-11-20T07:34:02.868183+0000 mon.smithi071 ( 2021-11-20T07:34:04.285 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T07:34:04.285 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:34:04.285 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: -20T07:34:02.881543+0000 2021-11-20T07:34:04.286 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: mon.smithi071 (mon.0) 73 : cluster [INF] 2021-11-20T07:34:04.286 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: Manager daemon smithi071.ynqmwn is now available 2021-11-20T07:34:04.286 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T07:34:04.287 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 2021-11-20T07:34:02.981208+0000 mon.smithi071 2021-11-20T07:34:04.287 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:34:04.287 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:34:04.287 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 11-20T07:34:02.982622+0000 2021-11-20T07:34:04.287 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: mon.smithi071 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:34:04.288 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 2021-11-20T07:34:04.288 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021 2021-11-20T07:34:04.288 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: -11-20T07:34:02.998655+0000 mon.smithi071 (mon.0) 2021-11-20T07:34:04.288 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 76 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:34:04.289 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:34:04.289 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: 07:34:03.006963+0000 mon.smithi071 (mon 2021-11-20T07:34:04.289 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:03 smithi071 conmon[33687]: .0) 77 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:34:04.901 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.898+0000 7f0871bc9700 1 Processor -- start 2021-11-20T07:34:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.898+0000 7f0871bc9700 1 -- start start 2021-11-20T07:34:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f0871bc9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f0871bc9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0fb3e0 con 0x7f086c0faad0 2021-11-20T07:34:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f086b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f086b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47060/0 (socket says 172.21.15.71:47060) 2021-11-20T07:34:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f086b7fe700 1 -- 172.21.15.71:0/1796335010 learned_addr learned my addr 172.21.15.71:0/1796335010 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f086b7fe700 1 -- 172.21.15.71:0/1796335010 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0fb520 con 0x7f086c0faad0 2021-11-20T07:34:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.899+0000 7f086b7fe700 1 --2- 172.21.15.71:0/1796335010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f086000b5f0 tx=0x7f0860006ee0).ready entity=mon.0 client_cookie=ba74531bec2a268a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:04.904 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f086a7fc700 1 -- 172.21.15.71:0/1796335010 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086000d600 con 0x7f086c0faad0 2021-11-20T07:34:04.904 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f086a7fc700 1 -- 172.21.15.71:0/1796335010 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f086000dc10 con 0x7f086c0faad0 2021-11-20T07:34:04.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f086a7fc700 1 -- 172.21.15.71:0/1796335010 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0860011cd0 con 0x7f086c0faad0 2021-11-20T07:34:04.908 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 -- 172.21.15.71:0/1796335010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 msgr2=0x7f086c0faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:04.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 --2- 172.21.15.71:0/1796335010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f086000b5f0 tx=0x7f0860006ee0).stop 2021-11-20T07:34:04.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 -- 172.21.15.71:0/1796335010 shutdown_connections 2021-11-20T07:34:04.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 --2- 172.21.15.71:0/1796335010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c0faef0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:04.909 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 -- 172.21.15.71:0/1796335010 >> 172.21.15.71:0/1796335010 conn(0x7f086c0f6310 msgr2=0x7f086c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:04.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 -- 172.21.15.71:0/1796335010 shutdown_connections 2021-11-20T07:34:04.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.900+0000 7f0871bc9700 1 -- 172.21.15.71:0/1796335010 wait complete. 2021-11-20T07:34:04.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.901+0000 7f0871bc9700 1 Processor -- start 2021-11-20T07:34:04.910 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.901+0000 7f0871bc9700 1 -- start start 2021-11-20T07:34:04.912 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.901+0000 7f0871bc9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:04.912 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.901+0000 7f0871bc9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c108cd0 con 0x7f086c0faad0 2021-11-20T07:34:04.912 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f086b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:04.912 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f086b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47062/0 (socket says 172.21.15.71:47062) 2021-11-20T07:34:04.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f086b7fe700 1 -- 172.21.15.71:0/3055426269 learned_addr learned my addr 172.21.15.71:0/3055426269 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:04.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f086b7fe700 1 -- 172.21.15.71:0/3055426269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086000b200 con 0x7f086c0faad0 2021-11-20T07:34:04.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f086b7fe700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f086000dfb0 tx=0x7f0860006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:04.913 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0860011df0 con 0x7f086c0faad0 2021-11-20T07:34:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c108ed0 con 0x7f086c0faad0 2021-11-20T07:34:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c1092e0 con 0x7f086c0faad0 2021-11-20T07:34:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0860006a70 con 0x7f086c0faad0 2021-11-20T07:34:04.914 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.902+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086001a720 con 0x7f086c0faad0 2021-11-20T07:34:04.915 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.903+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f086000f070 con 0x7f086c0faad0 2021-11-20T07:34:04.915 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.903+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c004fa0 con 0x7f086c0faad0 2021-11-20T07:34:04.915 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.903+0000 7f0853fff700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 0x7f0858034400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:04.915 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.904+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0860041e20 con 0x7f086c0faad0 2021-11-20T07:34:04.915 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.904+0000 7f086affd700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 0x7f0858034400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:04.916 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.904+0000 7f086affd700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 0x7f0858034400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f085c0096a0 tx=0x7f085c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:04.916 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:04.908+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0860010bc0 con 0x7f086c0faad0 2021-11-20T07:34:05.102 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.100+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 --> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f084c002a50 con 0x7f0858031f40 2021-11-20T07:34:05.106 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.104+0000 7f0853fff700 1 -- 172.21.15.71:0/3055426269 <== mgr.14142 v2:172.21.15.71:6800/4211885430 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f084c002a50 con 0x7f0858031f40 2021-11-20T07:34:05.106 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled grafana update... 2021-11-20T07:34:05.108 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 msgr2=0x7f0858034400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:05.108 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 0x7f0858034400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f085c0096a0 tx=0x7f085c006b40).stop 2021-11-20T07:34:05.109 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 msgr2=0x7f086c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:05.109 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f086000dfb0 tx=0x7f0860006ee0).stop 2021-11-20T07:34:05.109 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 shutdown_connections 2021-11-20T07:34:05.109 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f0858031f40 0x7f0858034400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:05.110 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 --2- 172.21.15.71:0/3055426269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f086c0faad0 0x7f086c1087e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:05.110 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.106+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 >> 172.21.15.71:0/3055426269 conn(0x7f086c0f6310 msgr2=0x7f086c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:05.110 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.107+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 shutdown_connections 2021-11-20T07:34:05.110 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:05.107+0000 7f0871bc9700 1 -- 172.21.15.71:0/3055426269 wait complete. 2021-11-20T07:34:05.136 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:04 smithi071 conmon[33687]: cluster 2021-11-20T07:34:03.871611+0000 mon.smithi071 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi071.ynqmwn(active, since 1.05918s) 2021-11-20T07:34:05.136 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:04 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:03.872578+0000 mgr.smithi071.ynqmwn (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2021-11-20T07:34:05.137 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:04 smithi071 conmon[33687]: audit 2021-11-20T07:34:03.874541+0000 mon.smithi071 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:05.137 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:04 smithi071 conmon[33687]: audit 2021-11-20T07:34:04.743972+0000 mon.smithi071 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying node-exporter service with default placement... 2021-11-20T07:34:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2021-11-20T07:34:06.479 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:06 smithi071 conmon[33687]: audit 2021-11-20T07:34:05.102560+0000 mgr.smithi071.ynqmwn (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:06.479 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:06 smithi071 conmon[33687]: cephadm 2021-11-20 2021-11-20T07:34:06.480 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:06 smithi071 conmon[33687]: T07:34:05.103087+0000 mgr.smithi071.ynqmwn (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2021-11-20T07:34:06.480 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:06 smithi071 conmon[33687]: audit 2021-11-20T07:34:05.105331+0000 mon.smithi071 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:06.480 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:06 smithi071 conmon[33687]: cluster 2021-11-20T07:34:05.745474+0000 mon.smithi071 (mon.0) 82 : cluster [DBG] mgrmap e11: smithi071.ynqmwn(active, since 2s) 2021-11-20T07:34:06.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.477+0000 7f289a197700 1 Processor -- start 2021-11-20T07:34:06.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.477+0000 7f289a197700 1 -- start start 2021-11-20T07:34:06.484 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.477+0000 7f289a197700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:06.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.477+0000 7f289a197700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28940fb480 con 0x7f28940faae0 2021-11-20T07:34:06.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.478+0000 7f28937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:06.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.478+0000 7f28937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47066/0 (socket says 172.21.15.71:47066) 2021-11-20T07:34:06.485 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.478+0000 7f28937fe700 1 -- 172.21.15.71:0/3906534637 learned_addr learned my addr 172.21.15.71:0/3906534637 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.478+0000 7f28937fe700 1 -- 172.21.15.71:0/3906534637 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28940fb5c0 con 0x7f28940faae0 2021-11-20T07:34:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.478+0000 7f28937fe700 1 --2- 172.21.15.71:0/3906534637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f288400b5f0 tx=0x7f2884006ee0).ready entity=mon.0 client_cookie=cb4e6b0c6575b825 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f2892ffd700 1 -- 172.21.15.71:0/3906534637 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f288400d600 con 0x7f28940faae0 2021-11-20T07:34:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f2892ffd700 1 -- 172.21.15.71:0/3906534637 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f288400dc10 con 0x7f28940faae0 2021-11-20T07:34:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f2892ffd700 1 -- 172.21.15.71:0/3906534637 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2884011cd0 con 0x7f28940faae0 2021-11-20T07:34:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 -- 172.21.15.71:0/3906534637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 msgr2=0x7f28940faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 --2- 172.21.15.71:0/3906534637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f288400b5f0 tx=0x7f2884006ee0).stop 2021-11-20T07:34:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 -- 172.21.15.71:0/3906534637 shutdown_connections 2021-11-20T07:34:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 --2- 172.21.15.71:0/3906534637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f28940faf00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 -- 172.21.15.71:0/3906534637 >> 172.21.15.71:0/3906534637 conn(0x7f28940f6280 msgr2=0x7f28940f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 -- 172.21.15.71:0/3906534637 shutdown_connections 2021-11-20T07:34:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.479+0000 7f289a197700 1 -- 172.21.15.71:0/3906534637 wait complete. 2021-11-20T07:34:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f289a197700 1 Processor -- start 2021-11-20T07:34:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f289a197700 1 -- start start 2021-11-20T07:34:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f289a197700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f289a197700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289410af10 con 0x7f28940faae0 2021-11-20T07:34:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f28937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f28937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47068/0 (socket says 172.21.15.71:47068) 2021-11-20T07:34:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f28937fe700 1 -- 172.21.15.71:0/3795043672 learned_addr learned my addr 172.21.15.71:0/3795043672 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.480+0000 7f28937fe700 1 -- 172.21.15.71:0/3795043672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288400b200 con 0x7f28940faae0 2021-11-20T07:34:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f28937fe700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2884006800 tx=0x7f2884006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2884011df0 con 0x7f28940faae0 2021-11-20T07:34:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f288401b8c0 con 0x7f28940faae0 2021-11-20T07:34:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289410b110 con 0x7f28940faae0 2021-11-20T07:34:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289410b520 con 0x7f28940faae0 2021-11-20T07:34:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.481+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f288401a800 con 0x7f28940faae0 2021-11-20T07:34:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.482+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f288400f070 con 0x7f28940faae0 2021-11-20T07:34:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.482+0000 7f2890ff9700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 0x7f28740323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.482+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2884041e10 con 0x7f28940faae0 2021-11-20T07:34:06.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.482+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2878004fa0 con 0x7f28940faae0 2021-11-20T07:34:06.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.482+0000 7f288affd700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 0x7f28740323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:06.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.483+0000 7f288affd700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 0x7f28740323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f287c0096a0 tx=0x7f287c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:06.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.486+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2884016070 con 0x7f28940faae0 2021-11-20T07:34:06.665 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.663+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 --> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f2878002a50 con 0x7f287402ff00 2021-11-20T07:34:06.668 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.667+0000 7f2890ff9700 1 -- 172.21.15.71:0/3795043672 <== mgr.14142 v2:172.21.15.71:6800/4211885430 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2878002a50 con 0x7f287402ff00 2021-11-20T07:34:06.669 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2021-11-20T07:34:06.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.668+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 msgr2=0x7f28740323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:06.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.668+0000 7f289a197700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 0x7f28740323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f287c0096a0 tx=0x7f287c006b40).stop 2021-11-20T07:34:06.671 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.668+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 msgr2=0x7f289410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:06.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.668+0000 7f289a197700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2884006800 tx=0x7f2884006ee0).stop 2021-11-20T07:34:06.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 shutdown_connections 2021-11-20T07:34:06.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f287402ff00 0x7f28740323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:06.672 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 --2- 172.21.15.71:0/3795043672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28940faae0 0x7f289410aa20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:06.673 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 >> 172.21.15.71:0/3795043672 conn(0x7f28940f6280 msgr2=0x7f289406f830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:06.674 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 shutdown_connections 2021-11-20T07:34:06.674 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:06.669+0000 7f289a197700 1 -- 172.21.15.71:0/3795043672 wait complete. 2021-11-20T07:34:07.072 INFO:teuthology.orchestra.run.smithi071.stderr:Deploying alertmanager service with default placement... 2021-11-20T07:34:07.072 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2021-11-20T07:34:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f4908851700 1 Processor -- start 2021-11-20T07:34:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f4908851700 1 -- start start 2021-11-20T07:34:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f4908851700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f4908851700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040fed60 con 0x7f490406fc90 2021-11-20T07:34:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f490259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f490259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47072/0 (socket says 172.21.15.71:47072) 2021-11-20T07:34:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.503+0000 7f490259c700 1 -- 172.21.15.71:0/2292119816 learned_addr learned my addr 172.21.15.71:0/2292119816 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.504+0000 7f490259c700 1 -- 172.21.15.71:0/2292119816 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040feea0 con 0x7f490406fc90 2021-11-20T07:34:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.504+0000 7f490259c700 1 --2- 172.21.15.71:0/2292119816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f48f400b5f0 tx=0x7f48f4006ee0).ready entity=mon.0 client_cookie=13375e62e34e46fb server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.504+0000 7f4901d9b700 1 -- 172.21.15.71:0/2292119816 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f400d600 con 0x7f490406fc90 2021-11-20T07:34:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.504+0000 7f4901d9b700 1 -- 172.21.15.71:0/2292119816 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48f400dc10 con 0x7f490406fc90 2021-11-20T07:34:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.504+0000 7f4901d9b700 1 -- 172.21.15.71:0/2292119816 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f4011cd0 con 0x7f490406fc90 2021-11-20T07:34:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 -- 172.21.15.71:0/2292119816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 msgr2=0x7f49040fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 --2- 172.21.15.71:0/2292119816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f48f400b5f0 tx=0x7f48f4006ee0).stop 2021-11-20T07:34:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 -- 172.21.15.71:0/2292119816 shutdown_connections 2021-11-20T07:34:07.511 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 --2- 172.21.15.71:0/2292119816 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f49040fe7e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:07.511 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 -- 172.21.15.71:0/2292119816 >> 172.21.15.71:0/2292119816 conn(0x7f49040f6280 msgr2=0x7f49040f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 -- 172.21.15.71:0/2292119816 shutdown_connections 2021-11-20T07:34:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.505+0000 7f4908851700 1 -- 172.21.15.71:0/2292119816 wait complete. 2021-11-20T07:34:07.514 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f4908851700 1 Processor -- start 2021-11-20T07:34:07.515 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f4908851700 1 -- start start 2021-11-20T07:34:07.515 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f4908851700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:07.515 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f4908851700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490410aed0 con 0x7f490406fc90 2021-11-20T07:34:07.515 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f490259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f490259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47074/0 (socket says 172.21.15.71:47074) 2021-11-20T07:34:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.506+0000 7f490259c700 1 -- 172.21.15.71:0/1121517024 learned_addr learned my addr 172.21.15.71:0/1121517024 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f490259c700 1 -- 172.21.15.71:0/1121517024 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f400b200 con 0x7f490406fc90 2021-11-20T07:34:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f490259c700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f48f4006cb0 tx=0x7f48f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f4011df0 con 0x7f490406fc90 2021-11-20T07:34:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490410b0d0 con 0x7f490406fc90 2021-11-20T07:34:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490410b4e0 con 0x7f490406fc90 2021-11-20T07:34:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48f401b8c0 con 0x7f490406fc90 2021-11-20T07:34:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.507+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f401a930 con 0x7f490406fc90 2021-11-20T07:34:07.518 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.508+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f48f400f070 con 0x7f490406fc90 2021-11-20T07:34:07.518 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.508+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904045ad0 con 0x7f490406fc90 2021-11-20T07:34:07.518 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.508+0000 7f48fb7fe700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 0x7f48e4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:07.518 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.508+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f48f4040fe0 con 0x7f490406fc90 2021-11-20T07:34:07.519 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.508+0000 7f48f9d9b700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 0x7f48e4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:07.519 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.509+0000 7f48f9d9b700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 0x7f48e4032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48ec0096a0 tx=0x7f48ec006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:07.519 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.512+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48f4019950 con 0x7f490406fc90 2021-11-20T07:34:07.693 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.691+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 --> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f490410be20 con 0x7f48e402feb0 2021-11-20T07:34:07.699 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.697+0000 7f48fb7fe700 1 -- 172.21.15.71:0/1121517024 <== mgr.14142 v2:172.21.15.71:6800/4211885430 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f490410be20 con 0x7f48e402feb0 2021-11-20T07:34:07.700 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2021-11-20T07:34:07.702 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 msgr2=0x7f48e4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:07.702 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 0x7f48e4032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48ec0096a0 tx=0x7f48ec006b40).stop 2021-11-20T07:34:07.702 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 msgr2=0x7f490410a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:07.703 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f48f4006cb0 tx=0x7f48f4006ee0).stop 2021-11-20T07:34:07.703 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 shutdown_connections 2021-11-20T07:34:07.703 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f48e402feb0 0x7f48e4032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:07.703 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 --2- 172.21.15.71:0/1121517024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f490406fc90 0x7f490410a9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:07.704 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 >> 172.21.15.71:0/1121517024 conn(0x7f49040f6280 msgr2=0x7f49040f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:07.704 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.699+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 shutdown_connections 2021-11-20T07:34:07.704 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:07.700+0000 7f4908851700 1 -- 172.21.15.71:0/1121517024 wait complete. 2021-11-20T07:34:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:07 smithi071 conmon[33687]: audit 2021-11-20T07:34:06.665353+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:34:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:07 smithi071 conmon[33687]: mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:07 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:06.666011+0000 mgr.smithi071.ynqmwn (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2021-11-20T07:34:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:07 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:34:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:07 smithi071 conmon[33687]: 11-20T07:34:06.668191+0000 mon.smithi071 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:08.106 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2021-11-20T07:34:08.564 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33c5d93700 1 Processor -- start 2021-11-20T07:34:08.564 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33c5d93700 1 -- start start 2021-11-20T07:34:08.564 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33c5d93700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:08.564 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33c5d93700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00fb3e0 con 0x7f33c00faad0 2021-11-20T07:34:08.565 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:08.565 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47078/0 (socket says 172.21.15.71:47078) 2021-11-20T07:34:08.565 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.559+0000 7f33bf7fe700 1 -- 172.21.15.71:0/2992761285 learned_addr learned my addr 172.21.15.71:0/2992761285 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:08.565 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.560+0000 7f33bf7fe700 1 -- 172.21.15.71:0/2992761285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00fb520 con 0x7f33c00faad0 2021-11-20T07:34:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.560+0000 7f33bf7fe700 1 --2- 172.21.15.71:0/2992761285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f33b000b5f0 tx=0x7f33b0006ee0).ready entity=mon.0 client_cookie=ac5492119733eb84 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.560+0000 7f33be7fc700 1 -- 172.21.15.71:0/2992761285 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b000d600 con 0x7f33c00faad0 2021-11-20T07:34:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.560+0000 7f33be7fc700 1 -- 172.21.15.71:0/2992761285 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33b000dc10 con 0x7f33c00faad0 2021-11-20T07:34:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.560+0000 7f33be7fc700 1 -- 172.21.15.71:0/2992761285 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b0011cd0 con 0x7f33c00faad0 2021-11-20T07:34:08.566 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- 172.21.15.71:0/2992761285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 msgr2=0x7f33c00faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:08.567 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 --2- 172.21.15.71:0/2992761285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f33b000b5f0 tx=0x7f33b0006ee0).stop 2021-11-20T07:34:08.567 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- 172.21.15.71:0/2992761285 shutdown_connections 2021-11-20T07:34:08.567 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 --2- 172.21.15.71:0/2992761285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c00faef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:08.567 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- 172.21.15.71:0/2992761285 >> 172.21.15.71:0/2992761285 conn(0x7f33c00f6310 msgr2=0x7f33c00f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:08.568 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- 172.21.15.71:0/2992761285 shutdown_connections 2021-11-20T07:34:08.568 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- 172.21.15.71:0/2992761285 wait complete. 2021-11-20T07:34:08.568 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 Processor -- start 2021-11-20T07:34:08.570 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.561+0000 7f33c5d93700 1 -- start start 2021-11-20T07:34:08.570 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33c5d93700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:08.570 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33c5d93700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00ff160 con 0x7f33c00faad0 2021-11-20T07:34:08.570 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:08.571 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47080/0 (socket says 172.21.15.71:47080) 2021-11-20T07:34:08.571 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33bf7fe700 1 -- 172.21.15.71:0/1441319795 learned_addr learned my addr 172.21.15.71:0/1441319795 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:08.572 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33bf7fe700 1 -- 172.21.15.71:0/1441319795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b000b200 con 0x7f33c00faad0 2021-11-20T07:34:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33bf7fe700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f33b000e040 tx=0x7f33b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b001b410 con 0x7f33c00faad0 2021-11-20T07:34:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33b001ba20 con 0x7f33c00faad0 2021-11-20T07:34:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c00ff300 con 0x7f33c00faad0 2021-11-20T07:34:08.574 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b001a910 con 0x7f33c00faad0 2021-11-20T07:34:08.574 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.562+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c00ff710 con 0x7f33c00faad0 2021-11-20T07:34:08.574 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f33b000f070 con 0x7f33c00faad0 2021-11-20T07:34:08.574 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33b7fff700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 0x7f33a0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:08.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c0045ad0 con 0x7f33c00faad0 2021-11-20T07:34:08.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f33b0046630 con 0x7f33c00faad0 2021-11-20T07:34:08.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33beffd700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 0x7f33a0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:08.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.564+0000 7f33beffd700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 0x7f33a0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f33a80096a0 tx=0x7f33a8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:08.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.568+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33b0016070 con 0x7f33c00faad0 2021-11-20T07:34:08.720 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.717+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f33c00fc840 con 0x7f33c00faad0 2021-11-20T07:34:08.724 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.723+0000 7f33b7fff700 1 -- 172.21.15.71:0/1441319795 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f33b0047060 con 0x7f33c00faad0 2021-11-20T07:34:08.727 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 msgr2=0x7f33a0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:08.728 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 0x7f33a0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f33a80096a0 tx=0x7f33a8006b40).stop 2021-11-20T07:34:08.728 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 msgr2=0x7f33c0100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:08.728 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f33b000e040 tx=0x7f33b0006ee0).stop 2021-11-20T07:34:08.728 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 shutdown_connections 2021-11-20T07:34:08.729 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f33a002fe60 0x7f33a0032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:08.729 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 --2- 172.21.15.71:0/1441319795 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33c00faad0 0x7f33c0100ce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:08.729 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.724+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 >> 172.21.15.71:0/1441319795 conn(0x7f33c00f6310 msgr2=0x7f33c00f6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:08.729 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.725+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 shutdown_connections 2021-11-20T07:34:08.729 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:08.725+0000 7f33c5d93700 1 -- 172.21.15.71:0/1441319795 wait complete. 2021-11-20T07:34:08.817 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:08 smithi071 conmon[33687]: audit 2021-11-20T07:34:07.693599+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:34:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:08 smithi071 conmon[33687]: mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:08 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:07.694134+0000 mgr.smithi071.ynqmwn (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2021-11-20T07:34:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:08 smithi071 conmon[33687]: audit 2021-11-20T07:34:07.698741+0000 mon.smithi071 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.71:0/1845384539' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:09.040 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-11-20T07:34:09.488 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62ffff700 1 Processor -- start 2021-11-20T07:34:09.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62ffff700 1 -- start start 2021-11-20T07:34:09.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:09.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62ffff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300720f0 con 0x7fd630071cd0 2021-11-20T07:34:09.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:09.489 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47084/0 (socket says 172.21.15.71:47084) 2021-11-20T07:34:09.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62effd700 1 -- 172.21.15.71:0/948779468 learned_addr learned my addr 172.21.15.71:0/948779468 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:09.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62effd700 1 -- 172.21.15.71:0/948779468 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630070760 con 0x7fd630071cd0 2021-11-20T07:34:09.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.483+0000 7fd62effd700 1 --2- 172.21.15.71:0/948779468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd62000b5f0 tx=0x7fd620006ee0).ready entity=mon.0 client_cookie=f4c0bc916038c3c0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:09.490 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62dffb700 1 -- 172.21.15.71:0/948779468 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd62000d600 con 0x7fd630071cd0 2021-11-20T07:34:09.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62dffb700 1 -- 172.21.15.71:0/948779468 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd62000dc10 con 0x7fd630071cd0 2021-11-20T07:34:09.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62dffb700 1 -- 172.21.15.71:0/948779468 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620011cd0 con 0x7fd630071cd0 2021-11-20T07:34:09.491 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 -- 172.21.15.71:0/948779468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 msgr2=0x7fd630070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 --2- 172.21.15.71:0/948779468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd62000b5f0 tx=0x7fd620006ee0).stop 2021-11-20T07:34:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 -- 172.21.15.71:0/948779468 shutdown_connections 2021-11-20T07:34:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 --2- 172.21.15.71:0/948779468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd630070270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:09.492 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 -- 172.21.15.71:0/948779468 >> 172.21.15.71:0/948779468 conn(0x7fd6300f71e0 msgr2=0x7fd6300f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:09.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 -- 172.21.15.71:0/948779468 shutdown_connections 2021-11-20T07:34:09.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.484+0000 7fd62ffff700 1 -- 172.21.15.71:0/948779468 wait complete. 2021-11-20T07:34:09.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62ffff700 1 Processor -- start 2021-11-20T07:34:09.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62ffff700 1 -- start start 2021-11-20T07:34:09.493 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62ffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:09.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62ffff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63010cec0 con 0x7fd630071cd0 2021-11-20T07:34:09.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:09.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47086/0 (socket says 172.21.15.71:47086) 2021-11-20T07:34:09.494 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62effd700 1 -- 172.21.15.71:0/2215553965 learned_addr learned my addr 172.21.15.71:0/2215553965 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:09.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.485+0000 7fd62effd700 1 -- 172.21.15.71:0/2215553965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62000b200 con 0x7fd630071cd0 2021-11-20T07:34:09.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd62effd700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd62000b1d0 tx=0x7fd620006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:09.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd620011df0 con 0x7fd630071cd0 2021-11-20T07:34:09.495 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd62001b8c0 con 0x7fd630071cd0 2021-11-20T07:34:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63010d0c0 con 0x7fd630071cd0 2021-11-20T07:34:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd62001a830 con 0x7fd630071cd0 2021-11-20T07:34:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.486+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63010d4d0 con 0x7fd630071cd0 2021-11-20T07:34:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.487+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd62000f070 con 0x7fd630071cd0 2021-11-20T07:34:09.496 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.487+0000 7fd6177fe700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 0x7fd61c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:09.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.487+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd620040f20 con 0x7fd630071cd0 2021-11-20T07:34:09.497 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.487+0000 7fd62e7fc700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 0x7fd61c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:09.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.487+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd630059100 con 0x7fd630071cd0 2021-11-20T07:34:09.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.488+0000 7fd62e7fc700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 0x7fd61c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6240096a0 tx=0x7fd624006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:09.498 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.491+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd620016070 con 0x7fd630071cd0 2021-11-20T07:34:09.642 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.640+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd630045ad0 con 0x7fd630071cd0 2021-11-20T07:34:09.644 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.643+0000 7fd6177fe700 1 -- 172.21.15.71:0/2215553965 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd6200108d0 con 0x7fd630071cd0 2021-11-20T07:34:09.647 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 msgr2=0x7fd61c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:09.647 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 0x7fd61c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6240096a0 tx=0x7fd624006b40).stop 2021-11-20T07:34:09.648 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 msgr2=0x7fd63010c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:09.648 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd62000b1d0 tx=0x7fd620006ee0).stop 2021-11-20T07:34:09.648 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 shutdown_connections 2021-11-20T07:34:09.648 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fd61c02feb0 0x7fd61c032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:09.648 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 --2- 172.21.15.71:0/2215553965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd630071cd0 0x7fd63010c9d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 >> 172.21.15.71:0/2215553965 conn(0x7fd6300f71e0 msgr2=0x7fd6300f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 shutdown_connections 2021-11-20T07:34:09.649 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:09.645+0000 7fd62ffff700 1 -- 172.21.15.71:0/2215553965 wait complete. 2021-11-20T07:34:09.788 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:09 smithi071 conmon[33687]: audit 2021-11-20T07:34:08.724168+0000 mon.smithi071 (mon.0) 2021-11-20T07:34:09.788 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:09 smithi071 conmon[33687]: 85 : audit [INF] from='client.? 172.21.15.71:0/1441319795' entity='client.admin' 2021-11-20T07:34:09.789 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:09 smithi071 conmon[33687]: audit 2021-11-20T07:34:09.644769+0000 mon.smithi071 (mon.0) 86 : audit [INF] from='client.? 172.21.15.71:0/2215553965' entity='client.admin' 2021-11-20T07:34:09.975 INFO:teuthology.orchestra.run.smithi071.stderr:Enabling the dashboard module... 2021-11-20T07:34:09.975 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2021-11-20T07:34:10.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.376+0000 7fec231b1700 1 Processor -- start 2021-11-20T07:34:10.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.376+0000 7fec231b1700 1 -- start start 2021-11-20T07:34:10.381 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.376+0000 7fec231b1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:10.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.376+0000 7fec231b1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0fc660 con 0x7fec1c06fc90 2021-11-20T07:34:10.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec20f4d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:10.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec20f4d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47090/0 (socket says 172.21.15.71:47090) 2021-11-20T07:34:10.382 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec20f4d700 1 -- 172.21.15.71:0/1755486424 learned_addr learned my addr 172.21.15.71:0/1755486424 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:10.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec20f4d700 1 -- 172.21.15.71:0/1755486424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c1010d0 con 0x7fec1c06fc90 2021-11-20T07:34:10.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec20f4d700 1 --2- 172.21.15.71:0/1755486424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fec0c00b5f0 tx=0x7fec0c006ee0).ready entity=mon.0 client_cookie=539bd220e7958d7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:10.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec137fe700 1 -- 172.21.15.71:0/1755486424 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c00d600 con 0x7fec1c06fc90 2021-11-20T07:34:10.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec137fe700 1 -- 172.21.15.71:0/1755486424 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec0c00dc10 con 0x7fec1c06fc90 2021-11-20T07:34:10.383 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.377+0000 7fec137fe700 1 -- 172.21.15.71:0/1755486424 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c011cd0 con 0x7fec1c06fc90 2021-11-20T07:34:10.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 -- 172.21.15.71:0/1755486424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 msgr2=0x7fec1c100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:10.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 --2- 172.21.15.71:0/1755486424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fec0c00b5f0 tx=0x7fec0c006ee0).stop 2021-11-20T07:34:10.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 -- 172.21.15.71:0/1755486424 shutdown_connections 2021-11-20T07:34:10.384 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 --2- 172.21.15.71:0/1755486424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c100be0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:10.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 -- 172.21.15.71:0/1755486424 >> 172.21.15.71:0/1755486424 conn(0x7fec1c0f6280 msgr2=0x7fec1c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:10.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 -- 172.21.15.71:0/1755486424 shutdown_connections 2021-11-20T07:34:10.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.378+0000 7fec231b1700 1 -- 172.21.15.71:0/1755486424 wait complete. 2021-11-20T07:34:10.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec231b1700 1 Processor -- start 2021-11-20T07:34:10.385 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec231b1700 1 -- start start 2021-11-20T07:34:10.386 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec231b1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:10.386 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec231b1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c10d0d0 con 0x7fec1c06fc90 2021-11-20T07:34:10.386 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec20f4d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:10.386 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec20f4d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47092/0 (socket says 172.21.15.71:47092) 2021-11-20T07:34:10.387 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec20f4d700 1 -- 172.21.15.71:0/1463396028 learned_addr learned my addr 172.21.15.71:0/1463396028 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:10.387 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.379+0000 7fec20f4d700 1 -- 172.21.15.71:0/1463396028 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c00b200 con 0x7fec1c06fc90 2021-11-20T07:34:10.388 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec20f4d700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fec0c00b1d0 tx=0x7fec0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:10.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c011df0 con 0x7fec1c06fc90 2021-11-20T07:34:10.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c10d2d0 con 0x7fec1c06fc90 2021-11-20T07:34:10.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec0c01b8c0 con 0x7fec1c06fc90 2021-11-20T07:34:10.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c01a7b0 con 0x7fec1c06fc90 2021-11-20T07:34:10.389 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.380+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c10d6e0 con 0x7fec1c06fc90 2021-11-20T07:34:10.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fec0c00f070 con 0x7fec1c06fc90 2021-11-20T07:34:10.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec117fa700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 0x7fec08032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:10.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fec0c0421e0 con 0x7fec1c06fc90 2021-11-20T07:34:10.390 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec13fff700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 0x7fec08032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:10.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c059100 con 0x7fec1c06fc90 2021-11-20T07:34:10.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.381+0000 7fec13fff700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 0x7fec08032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fec180096a0 tx=0x7fec18006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:10.391 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.385+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0c016070 con 0x7fec1c06fc90 2021-11-20T07:34:10.573 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:10.571+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fec1c045ad0 con 0x7fec1c06fc90 2021-11-20T07:34:11.312 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.310+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fec0c0257e0 con 0x7fec1c06fc90 2021-11-20T07:34:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.312+0000 7fec117fa700 1 -- 172.21.15.71:0/1463396028 <== mon.0 v2:172.21.15.71:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7fec0c0239b0 con 0x7fec1c06fc90 2021-11-20T07:34:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 msgr2=0x7fec08032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 0x7fec08032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fec180096a0 tx=0x7fec18006b40).stop 2021-11-20T07:34:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 msgr2=0x7fec1c10cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:11.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fec0c00b1d0 tx=0x7fec0c006ee0).stop 2021-11-20T07:34:11.318 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 shutdown_connections 2021-11-20T07:34:11.318 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fec0802feb0 0x7fec08032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:11.318 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 --2- 172.21.15.71:0/1463396028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec1c06fc90 0x7fec1c10cbe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:11.318 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 >> 172.21.15.71:0/1463396028 conn(0x7fec1c0f6280 msgr2=0x7fec1c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:11.319 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.314+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 shutdown_connections 2021-11-20T07:34:11.319 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:11.315+0000 7fec231b1700 1 -- 172.21.15.71:0/1463396028 wait complete. 2021-11-20T07:34:11.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:11 smithi071 conmon[33687]: audit 2021-11-20T07:34:10.573156+0000 mon.smithi071 (mon.0) 87 : 2021-11-20T07:34:11.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:11 smithi071 conmon[33687]: audit [INF] from='client.? 172.21.15.71:0/1463396028' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-11-20T07:34:11.715 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T07:34:12.271 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.268+0000 7f4333a8e700 1 Processor -- start 2021-11-20T07:34:12.271 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.268+0000 7f4333a8e700 1 -- start start 2021-11-20T07:34:12.272 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.268+0000 7f4333a8e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:12.272 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.268+0000 7f4333a8e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0fed20 con 0x7f432c06fcd0 2021-11-20T07:34:12.272 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f433182a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:12.272 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f433182a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47106/0 (socket says 172.21.15.71:47106) 2021-11-20T07:34:12.273 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f433182a700 1 -- 172.21.15.71:0/199140280 learned_addr learned my addr 172.21.15.71:0/199140280 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:12.273 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f433182a700 1 -- 172.21.15.71:0/199140280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c0fee60 con 0x7f432c06fcd0 2021-11-20T07:34:12.273 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f433182a700 1 --2- 172.21.15.71:0/199140280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f431c00b5f0 tx=0x7f431c006ee0).ready entity=mon.0 client_cookie=693bbfa47e58473d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:12.273 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f4330828700 1 -- 172.21.15.71:0/199140280 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c00d600 con 0x7f432c06fcd0 2021-11-20T07:34:12.274 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f4330828700 1 -- 172.21.15.71:0/199140280 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f431c00dc10 con 0x7f432c06fcd0 2021-11-20T07:34:12.274 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.269+0000 7f4330828700 1 -- 172.21.15.71:0/199140280 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c011cd0 con 0x7f432c06fcd0 2021-11-20T07:34:12.276 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 -- 172.21.15.71:0/199140280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 msgr2=0x7f432c0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:12.276 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 --2- 172.21.15.71:0/199140280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f431c00b5f0 tx=0x7f431c006ee0).stop 2021-11-20T07:34:12.277 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 -- 172.21.15.71:0/199140280 shutdown_connections 2021-11-20T07:34:12.277 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 --2- 172.21.15.71:0/199140280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fe7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:12.277 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 -- 172.21.15.71:0/199140280 >> 172.21.15.71:0/199140280 conn(0x7f432c0f6280 msgr2=0x7f432c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:12.277 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 -- 172.21.15.71:0/199140280 shutdown_connections 2021-11-20T07:34:12.278 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.270+0000 7f4333a8e700 1 -- 172.21.15.71:0/199140280 wait complete. 2021-11-20T07:34:12.278 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f4333a8e700 1 Processor -- start 2021-11-20T07:34:12.278 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f4333a8e700 1 -- start start 2021-11-20T07:34:12.278 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f4333a8e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:12.279 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f4333a8e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0fa6a0 con 0x7f432c06fcd0 2021-11-20T07:34:12.279 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f433182a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:12.279 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f433182a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47108/0 (socket says 172.21.15.71:47108) 2021-11-20T07:34:12.279 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f433182a700 1 -- 172.21.15.71:0/3598761290 learned_addr learned my addr 172.21.15.71:0/3598761290 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:12.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f433182a700 1 -- 172.21.15.71:0/3598761290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c00b200 con 0x7f432c06fcd0 2021-11-20T07:34:12.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f433182a700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f431c00dfb0 tx=0x7f431c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:12.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c011df0 con 0x7f432c06fcd0 2021-11-20T07:34:12.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c0fa840 con 0x7f432c06fcd0 2021-11-20T07:34:12.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f431c01b8c0 con 0x7f432c06fcd0 2021-11-20T07:34:12.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.271+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c01a800 con 0x7f432c06fcd0 2021-11-20T07:34:12.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.272+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c0fac50 con 0x7f432c06fcd0 2021-11-20T07:34:12.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.272+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c045ad0 con 0x7f432c06fcd0 2021-11-20T07:34:12.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.273+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f431c00f070 con 0x7f432c06fcd0 2021-11-20T07:34:12.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.273+0000 7f431a7fc700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 0x7f4314032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:12.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.273+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f431c040ff0 con 0x7f432c06fcd0 2021-11-20T07:34:12.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.273+0000 7f4331029700 1 -- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 msgr2=0x7f4314032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:12.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.273+0000 7f4331029700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 0x7f4314032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:34:12.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.277+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f431c019950 con 0x7f432c06fcd0 2021-11-20T07:34:12.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.471+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f432c110a60 con 0x7f432c06fcd0 2021-11-20T07:34:12.474 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.471+0000 7f431a7fc700 1 -- 172.21.15.71:0/3598761290 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f431c016070 con 0x7f432c06fcd0 2021-11-20T07:34:12.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:34:12.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "epoch": 12, 2021-11-20T07:34:12.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "available": true, 2021-11-20T07:34:12.475 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "active_name": "smithi071.ynqmwn", 2021-11-20T07:34:12.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T07:34:12.476 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:34:12.477 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 msgr2=0x7f4314032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:34:12.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 0x7f4314032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:12.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 msgr2=0x7f432c0fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:12.478 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f431c00dfb0 tx=0x7f431c006ee0).stop 2021-11-20T07:34:12.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 shutdown_connections 2021-11-20T07:34:12.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7f431402feb0 0x7f4314032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:12.479 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 --2- 172.21.15.71:0/3598761290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f432c06fcd0 0x7f432c0fc220 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:12.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 >> 172.21.15.71:0/3598761290 conn(0x7f432c0f6280 msgr2=0x7f432c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:12.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 shutdown_connections 2021-11-20T07:34:12.480 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:12.473+0000 7f4333a8e700 1 -- 172.21.15.71:0/3598761290 wait complete. 2021-11-20T07:34:12.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:12 smithi071 conmon[33687]: audit 2021-11-20T07:34:11.313501+0000 mon.smithi071 (mon.0) 88 : audit [INF] 2021-11-20T07:34:12.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:12 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/1463396028' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-11-20T07:34:12.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:12 smithi071 conmon[33687]: cluster 2021-11-20T07:34:11.313590+0000 mon.smithi071 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi071.ynqmwn(active, since 8s) 2021-11-20T07:34:12.750 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for the mgr to restart... 2021-11-20T07:34:12.751 INFO:teuthology.orchestra.run.smithi071.stderr:Waiting for mgr epoch 12... 2021-11-20T07:34:12.751 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T07:34:13.169 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.165+0000 7fdf835d4700 1 Processor -- start 2021-11-20T07:34:13.169 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.165+0000 7fdf835d4700 1 -- start start 2021-11-20T07:34:13.169 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf835d4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:13.169 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf835d4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0fb3e0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.170 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf81370700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:13.170 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf81370700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47114/0 (socket says 172.21.15.71:47114) 2021-11-20T07:34:13.170 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf81370700 1 -- 172.21.15.71:0/2719744138 learned_addr learned my addr 172.21.15.71:0/2719744138 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf81370700 1 -- 172.21.15.71:0/2719744138 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0fb520 con 0x7fdf7c0faad0 2021-11-20T07:34:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf81370700 1 --2- 172.21.15.71:0/2719744138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fdf74009960 tx=0x7fdf740092d0).ready entity=mon.0 client_cookie=7af9db22e75e0781 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf6bfff700 1 -- 172.21.15.71:0/2719744138 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7400b9c0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.166+0000 7fdf6bfff700 1 -- 172.21.15.71:0/2719744138 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf74011400 con 0x7fdf7c0faad0 2021-11-20T07:34:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf6bfff700 1 -- 172.21.15.71:0/2719744138 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf74010580 con 0x7fdf7c0faad0 2021-11-20T07:34:13.172 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 -- 172.21.15.71:0/2719744138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 msgr2=0x7fdf7c0faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:13.172 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2719744138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fdf74009960 tx=0x7fdf740092d0).stop 2021-11-20T07:34:13.172 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 -- 172.21.15.71:0/2719744138 shutdown_connections 2021-11-20T07:34:13.172 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2719744138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c0faef0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:13.173 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 -- 172.21.15.71:0/2719744138 >> 172.21.15.71:0/2719744138 conn(0x7fdf7c0f6310 msgr2=0x7fdf7c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:13.175 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 -- 172.21.15.71:0/2719744138 shutdown_connections 2021-11-20T07:34:13.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.167+0000 7fdf835d4700 1 -- 172.21.15.71:0/2719744138 wait complete. 2021-11-20T07:34:13.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf835d4700 1 Processor -- start 2021-11-20T07:34:13.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf835d4700 1 -- start start 2021-11-20T07:34:13.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf835d4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:13.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf835d4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c108c70 con 0x7fdf7c0faad0 2021-11-20T07:34:13.177 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf81370700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:13.177 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf81370700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47116/0 (socket says 172.21.15.71:47116) 2021-11-20T07:34:13.177 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.168+0000 7fdf81370700 1 -- 172.21.15.71:0/2465120062 learned_addr learned my addr 172.21.15.71:0/2465120062 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:13.177 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf81370700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74009570 con 0x7fdf7c0faad0 2021-11-20T07:34:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf81370700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdf74011cf0 tx=0x7fdf740092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf74011aa0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c108e70 con 0x7fdf7c0faad0 2021-11-20T07:34:13.178 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf74010cf0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.179 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7401bdd0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.179 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.169+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c109280 con 0x7fdf7c0faad0 2021-11-20T07:34:13.179 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdf7400f070 con 0x7fdf7c0faad0 2021-11-20T07:34:13.179 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf69ffb700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:13.180 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf64034c80 con 0x7fdf64032100 2021-11-20T07:34:13.180 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdf74041ba0 con 0x7fdf7c0faad0 2021-11-20T07:34:13.180 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf80b6f700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:13.180 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.170+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:34:13.372 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.371+0000 7fdf80b6f700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:13.373 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.371+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T07:34:13.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:13 smithi071 conmon[33687]: audit 2021-11-20T07:34:12.472925+0000 mon.smithi071 (mon.0) 90 : audit 2021-11-20T07:34:13.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:13 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/3598761290' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T07:34:13.773 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.772+0000 7fdf80b6f700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:13.774 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:13.772+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T07:34:14.574 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:14.573+0000 7fdf80b6f700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:14.575 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:14.573+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T07:34:16.176 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:16.174+0000 7fdf80b6f700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6800/4211885430 2021-11-20T07:34:16.177 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:16.174+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T07:34:16.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:16.315+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fdf74024920 con 0x7fdf7c0faad0 2021-11-20T07:34:16.317 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:16.315+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 msgr2=0x7fdf640345c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:34:16.318 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:16.315+0000 7fdf69ffb700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:16.460 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:16 smithi071 conmon[33687]: cluster 2021-11-20T07:34:16.294053+0000 mon.smithi071 (mon.0) 91 : cluster [INF] Active manager daemon smithi071.ynqmwn restarted 2021-11-20T07:34:16.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:16 smithi071 conmon[33687]: cluster 2021-11-20T07:34:16.294468+0000 mon.smithi071 (mon.0) 92 : cluster [INF] Activating manager daemon smithi071.ynqmwn 2021-11-20T07:34:16.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:16 smithi071 conmon[33687]: cluster 2021-11-20T07:34:16.298592+0000 mon.smithi071 (mon.0 2021-11-20T07:34:16.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:16 smithi071 conmon[33687]: ) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-11-20T07:34:17.325 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.321+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fdf74042fa0 con 0x7fdf7c0faad0 2021-11-20T07:34:17.325 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.321+0000 7fdf69ffb700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 0x7fdf64038180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:17.326 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.322+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf64034c80 con 0x7fdf64035d90 2021-11-20T07:34:17.326 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.322+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 0x7fdf64038180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:17.326 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.322+0000 7fdf80b6f700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 0x7fdf64038180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdf6c006fd0 tx=0x7fdf6c006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:17.326 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.323+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fdf64034c80 con 0x7fdf64035d90 2021-11-20T07:34:17.332 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdf7c059100 con 0x7fdf64035d90 2021-11-20T07:34:17.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf69ffb700 1 -- 172.21.15.71:0/2465120062 <== mgr.14162 v2:172.21.15.71:6800/429188327 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fdf7c059100 con 0x7fdf64035d90 2021-11-20T07:34:17.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: { 2021-11-20T07:34:17.333 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2021-11-20T07:34:17.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T07:34:17.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: } 2021-11-20T07:34:17.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 msgr2=0x7fdf64038180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:17.334 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 0x7fdf64038180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdf6c006fd0 tx=0x7fdf6c006cf0).stop 2021-11-20T07:34:17.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 msgr2=0x7fdf7c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:17.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.329+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdf74011cf0 tx=0x7fdf740092d0).stop 2021-11-20T07:34:17.335 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 shutdown_connections 2021-11-20T07:34:17.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdf64035d90 0x7fdf64038180 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:17.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf7c0faad0 0x7fdf7c108780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:17.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 --2- 172.21.15.71:0/2465120062 >> [v2:172.21.15.71:6800/4211885430,v1:172.21.15.71:6801/4211885430] conn(0x7fdf64032100 0x7fdf640345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:34:17.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 >> 172.21.15.71:0/2465120062 conn(0x7fdf7c0f6310 msgr2=0x7fdf7c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:17.336 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 shutdown_connections 2021-11-20T07:34:17.337 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:17.330+0000 7fdf835d4700 1 -- 172.21.15.71:0/2465120062 wait complete. 2021-11-20T07:34:17.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: cluster 2021-11-20T07:34:16.317490+0000 mon.smithi071 (mon. 2021-11-20T07:34:17.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: 0) 94 : cluster [DBG] mgrmap e13: smithi071.ynqmwn(active, starting, since 0.02311s) 2021-11-20T07:34:17.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.318568+0000 mon.smithi071 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:34:17.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.319106+0000 mon.smithi071 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi071.ynqmwn", "id": "smithi071.ynqmwn"}]: dispatch 2021-11-20T07:34:17.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.319683+0000 mon.smithi071 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T07:34:17.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.319830+0000 mon.smithi071 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T07:34:17.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.319971+0000 mon.smithi071 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T07:34:17.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: cluster 2021-11-20T07:34:16.333770+0000 mon.smithi071 (mon.0) 100 : cluster [INF] Manager daemon smithi071.ynqmwn is now available 2021-11-20T07:34:17.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.433492+0000 mon.smithi071 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:34:17.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.435500+0000 mon.smithi071 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:34:17.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.451278+0000 mon.smithi071 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:34:17.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:17 smithi071 conmon[33687]: audit 2021-11-20T07:34:16.453653+0000 mon.smithi071 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:34:17.594 INFO:teuthology.orchestra.run.smithi071.stderr:mgr epoch 12 is available 2021-11-20T07:34:17.595 INFO:teuthology.orchestra.run.smithi071.stderr:Generating a dashboard self-signed certificate... 2021-11-20T07:34:17.595 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2021-11-20T07:34:18.280 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.277+0000 7f4e24879700 1 Processor -- start 2021-11-20T07:34:18.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.277+0000 7f4e24879700 1 -- start start 2021-11-20T07:34:18.281 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e24879700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:18.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e24879700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200fed20 con 0x7f4e2006fcd0 2021-11-20T07:34:18.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:18.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47160/0 (socket says 172.21.15.71:47160) 2021-11-20T07:34:18.282 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e1e59c700 1 -- 172.21.15.71:0/1280218527 learned_addr learned my addr 172.21.15.71:0/1280218527 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:18.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e1e59c700 1 -- 172.21.15.71:0/1280218527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200fee60 con 0x7f4e2006fcd0 2021-11-20T07:34:18.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.278+0000 7f4e1e59c700 1 --2- 172.21.15.71:0/1280218527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f4e1400b5f0 tx=0x7f4e14006ee0).ready entity=mon.0 client_cookie=f82b502fe6918e92 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:18.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.279+0000 7f4e1d59a700 1 -- 172.21.15.71:0/1280218527 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e1400d600 con 0x7f4e2006fcd0 2021-11-20T07:34:18.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.279+0000 7f4e1d59a700 1 -- 172.21.15.71:0/1280218527 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e1400dc10 con 0x7f4e2006fcd0 2021-11-20T07:34:18.283 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.279+0000 7f4e1d59a700 1 -- 172.21.15.71:0/1280218527 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e14011cd0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.287 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.279+0000 7f4e24879700 1 -- 172.21.15.71:0/1280218527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 msgr2=0x7f4e200fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:18.287 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.279+0000 7f4e24879700 1 --2- 172.21.15.71:0/1280218527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f4e1400b5f0 tx=0x7f4e14006ee0).stop 2021-11-20T07:34:18.287 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- 172.21.15.71:0/1280218527 shutdown_connections 2021-11-20T07:34:18.288 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 --2- 172.21.15.71:0/1280218527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e200fe7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:18.288 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- 172.21.15.71:0/1280218527 >> 172.21.15.71:0/1280218527 conn(0x7f4e200f6280 msgr2=0x7f4e200f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:18.288 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- 172.21.15.71:0/1280218527 shutdown_connections 2021-11-20T07:34:18.288 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- 172.21.15.71:0/1280218527 wait complete. 2021-11-20T07:34:18.288 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 Processor -- start 2021-11-20T07:34:18.289 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- start start 2021-11-20T07:34:18.289 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:18.290 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e24879700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2010aee0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.290 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:18.290 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47162/0 (socket says 172.21.15.71:47162) 2021-11-20T07:34:18.290 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.280+0000 7f4e1e59c700 1 -- 172.21.15.71:0/2662463139 learned_addr learned my addr 172.21.15.71:0/2662463139 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:18.291 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e1e59c700 1 -- 172.21.15.71:0/2662463139 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1400b200 con 0x7f4e2006fcd0 2021-11-20T07:34:18.291 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e1e59c700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4e1400b1d0 tx=0x7f4e14006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:18.291 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e14011df0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.292 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e1401b8c0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.292 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2010b0e0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.292 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e1401a7b0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.292 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.281+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2010b4f0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.293 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.282+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f4e1400f070 con 0x7f4e2006fcd0 2021-11-20T07:34:18.293 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.282+0000 7f4e0effd700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 0x7f4e04032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:18.293 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.282+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e140412d0 con 0x7f4e2006fcd0 2021-11-20T07:34:18.293 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.282+0000 7f4e1dd9b700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 0x7f4e04032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:18.293 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.282+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e20059100 con 0x7f4e2006fcd0 2021-11-20T07:34:18.294 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.283+0000 7f4e1dd9b700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 0x7f4e04032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4e100096a0 tx=0x7f4e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:18.294 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.287+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e14016070 con 0x7f4e2006fcd0 2021-11-20T07:34:18.463 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.461+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e20104960 con 0x7f4e0402fd70 2021-11-20T07:34:18.508 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.506+0000 7f4e0effd700 1 -- 172.21.15.71:0/2662463139 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4e20104960 con 0x7f4e0402fd70 2021-11-20T07:34:18.509 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: Self-signed certificate created 2021-11-20T07:34:18.511 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.508+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 msgr2=0x7f4e04032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:18.511 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.508+0000 7f4e24879700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 0x7f4e04032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4e100096a0 tx=0x7f4e10006b40).stop 2021-11-20T07:34:18.511 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.508+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 msgr2=0x7f4e2010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:18.512 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.508+0000 7f4e24879700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4e1400b1d0 tx=0x7f4e14006ee0).stop 2021-11-20T07:34:18.512 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 shutdown_connections 2021-11-20T07:34:18.512 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e0402fd70 0x7f4e04032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:18.513 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 --2- 172.21.15.71:0/2662463139 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e2006fcd0 0x7f4e2010a9f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:18.513 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 >> 172.21.15.71:0/2662463139 conn(0x7f4e200f6280 msgr2=0x7f4e200f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:18.513 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 shutdown_connections 2021-11-20T07:34:18.513 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:18.509+0000 7f4e24879700 1 -- 172.21.15.71:0/2662463139 wait complete. 2021-11-20T07:34:18.534 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:18 smithi071 conmon[33687]: audit 2021-11-20T07:34:17.324100+0000 mgr.smithi071.ynqmwn (mgr.14162) 1 : audit 2021-11-20T07:34:18.534 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:18 smithi071 conmon[33687]: [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T07:34:18.534 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:18 smithi071 conmon[33687]: cluster 2021-11-20T07:34:17.324526+0000 mon.smithi071 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi071.ynqmwn(active, since 1.03014s) 2021-11-20T07:34:18.535 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:18 smithi071 conmon[33687]: audit 2021-11-20T07:34:17.330673+0000 mgr.smithi071.ynqmwn (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T07:34:18.535 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:18 smithi071 conmon[33687]: audit 2021-11-20T07:34:18.227844+0000 mon.smithi071 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:18.853 INFO:teuthology.orchestra.run.smithi071.stderr:Creating initial admin user... 2021-11-20T07:34:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfgzuegxb:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-11-20T07:34:19.428 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d74046700 1 Processor -- start 2021-11-20T07:34:19.429 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d74046700 1 -- start start 2021-11-20T07:34:19.429 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d74046700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:19.429 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d74046700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c0fb3e0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.429 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d71de2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:19.430 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d71de2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47166/0 (socket says 172.21.15.71:47166) 2021-11-20T07:34:19.430 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.426+0000 7f3d71de2700 1 -- 172.21.15.71:0/1366558406 learned_addr learned my addr 172.21.15.71:0/1366558406 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:19.430 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.427+0000 7f3d71de2700 1 -- 172.21.15.71:0/1366558406 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c0fb520 con 0x7f3d6c0faad0 2021-11-20T07:34:19.430 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.427+0000 7f3d71de2700 1 --2- 172.21.15.71:0/1366558406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3d6800b870 tx=0x7f3d68006f70).ready entity=mon.0 client_cookie=c0df54ae08e6daa0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:19.431 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.427+0000 7f3d70de0700 1 -- 172.21.15.71:0/1366558406 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6800da20 con 0x7f3d6c0faad0 2021-11-20T07:34:19.431 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.427+0000 7f3d70de0700 1 -- 172.21.15.71:0/1366558406 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d68010470 con 0x7f3d6c0faad0 2021-11-20T07:34:19.431 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.427+0000 7f3d70de0700 1 -- 172.21.15.71:0/1366558406 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6800f5b0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.435 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- 172.21.15.71:0/1366558406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 msgr2=0x7f3d6c0faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:19.435 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 --2- 172.21.15.71:0/1366558406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3d6800b870 tx=0x7f3d68006f70).stop 2021-11-20T07:34:19.435 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- 172.21.15.71:0/1366558406 shutdown_connections 2021-11-20T07:34:19.435 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 --2- 172.21.15.71:0/1366558406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c0faef0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:19.436 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- 172.21.15.71:0/1366558406 >> 172.21.15.71:0/1366558406 conn(0x7f3d6c0f6310 msgr2=0x7f3d6c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:19.436 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- 172.21.15.71:0/1366558406 shutdown_connections 2021-11-20T07:34:19.436 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- 172.21.15.71:0/1366558406 wait complete. 2021-11-20T07:34:19.436 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 Processor -- start 2021-11-20T07:34:19.437 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- start start 2021-11-20T07:34:19.437 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:19.437 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.428+0000 7f3d74046700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c108c90 con 0x7f3d6c0faad0 2021-11-20T07:34:19.437 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d71de2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:19.437 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d71de2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47168/0 (socket says 172.21.15.71:47168) 2021-11-20T07:34:19.438 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d71de2700 1 -- 172.21.15.71:0/4008402625 learned_addr learned my addr 172.21.15.71:0/4008402625 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:19.438 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d71de2700 1 -- 172.21.15.71:0/4008402625 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6800b480 con 0x7f3d6c0faad0 2021-11-20T07:34:19.438 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d71de2700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3d6800b840 tx=0x7f3d68006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:19.438 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6800f3d0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.439 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c108e90 con 0x7f3d6c0faad0 2021-11-20T07:34:19.439 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d6800fef0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.439 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d68019c70 con 0x7f3d6c0faad0 2021-11-20T07:34:19.439 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.429+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c1092a0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.440 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.430+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3d68016030 con 0x7f3d6c0faad0 2021-11-20T07:34:19.440 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.430+0000 7f3d5a7fc700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 0x7f3d54032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:19.440 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.430+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d68044750 con 0x7f3d6c0faad0 2021-11-20T07:34:19.440 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.430+0000 7f3d715e1700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 0x7f3d54032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:19.440 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.430+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c045ad0 con 0x7f3d6c0faad0 2021-11-20T07:34:19.441 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.431+0000 7f3d715e1700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 0x7f3d54032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3d5c0096a0 tx=0x7f3d5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:19.441 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.435+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d68024420 con 0x7f3d6c0faad0 2021-11-20T07:34:19.607 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.605+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3d6c0fc840 con 0x7f3d5402feb0 2021-11-20T07:34:19.817 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:19 smithi071 conmon[33687]: audit 2021-11-20T07:34:18.463358+0000 mgr.smithi071.ynqmwn (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:19 smithi071 conmon[33687]: audit 2021-11-20T07:34:18.505874+0000 mon.smithi071 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:19 smithi071 conmon[33687]: audit 2021-11-20T07:34:18.507699+0000 mon.smithi071 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:19 smithi071 conmon[33687]: cluster 2021-11-20T07:34:19.229549+0000 mon.smithi071 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi071.ynqmwn(active, since 2s) 2021-11-20T07:34:19.825 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.824+0000 7f3d5a7fc700 1 -- 172.21.15.71:0/4008402625 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3d6c0fc840 con 0x7f3d5402feb0 2021-11-20T07:34:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$8n.8o8gDK4bqZTReU/Yy7elzp5XH4TyGhd2BwFI1mu11yUymgAItq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1637393659, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-11-20T07:34:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 msgr2=0x7f3d54032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 0x7f3d54032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3d5c0096a0 tx=0x7f3d5c006b40).stop 2021-11-20T07:34:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 msgr2=0x7f3d6c1087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3d6800b840 tx=0x7f3d68006f70).stop 2021-11-20T07:34:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 shutdown_connections 2021-11-20T07:34:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d5402feb0 0x7f3d54032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 --2- 172.21.15.71:0/4008402625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d6c0faad0 0x7f3d6c1087a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 >> 172.21.15.71:0/4008402625 conn(0x7f3d6c0f6310 msgr2=0x7f3d6c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 shutdown_connections 2021-11-20T07:34:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:19.826+0000 7f3d74046700 1 -- 172.21.15.71:0/4008402625 wait complete. 2021-11-20T07:34:20.342 INFO:teuthology.orchestra.run.smithi071.stderr:Fetching dashboard port number... 2021-11-20T07:34:20.343 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi071 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpkzm4jljc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx2p_14lx:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2021-11-20T07:34:20.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c86c3e700 1 Processor -- start 2021-11-20T07:34:20.806 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c86c3e700 1 -- start start 2021-11-20T07:34:20.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c86c3e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:20.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c86c3e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800fb1b0 con 0x7f4c800fa8a0 2021-11-20T07:34:20.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c85c3c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:20.807 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c85c3c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47172/0 (socket says 172.21.15.71:47172) 2021-11-20T07:34:20.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.801+0000 7f4c85c3c700 1 -- 172.21.15.71:0/3583936401 learned_addr learned my addr 172.21.15.71:0/3583936401 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:20.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.802+0000 7f4c85c3c700 1 -- 172.21.15.71:0/3583936401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c800fb2f0 con 0x7f4c800fa8a0 2021-11-20T07:34:20.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.802+0000 7f4c85c3c700 1 --2- 172.21.15.71:0/3583936401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c7c00b5f0 tx=0x7f4c7c006ee0).ready entity=mon.0 client_cookie=9e3c6289a3e387bf server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:20.808 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.802+0000 7f4c84c3a700 1 -- 172.21.15.71:0/3583936401 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7c00d600 con 0x7f4c800fa8a0 2021-11-20T07:34:20.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.802+0000 7f4c84c3a700 1 -- 172.21.15.71:0/3583936401 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c7c00dc10 con 0x7f4c800fa8a0 2021-11-20T07:34:20.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.802+0000 7f4c84c3a700 1 -- 172.21.15.71:0/3583936401 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7c011cd0 con 0x7f4c800fa8a0 2021-11-20T07:34:20.809 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 -- 172.21.15.71:0/3583936401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 msgr2=0x7f4c800facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:20.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/3583936401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c7c00b5f0 tx=0x7f4c7c006ee0).stop 2021-11-20T07:34:20.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 -- 172.21.15.71:0/3583936401 shutdown_connections 2021-11-20T07:34:20.810 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/3583936401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c800facc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:20.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 -- 172.21.15.71:0/3583936401 >> 172.21.15.71:0/3583936401 conn(0x7f4c800f60e0 msgr2=0x7f4c800f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:20.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 -- 172.21.15.71:0/3583936401 shutdown_connections 2021-11-20T07:34:20.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.803+0000 7f4c86c3e700 1 -- 172.21.15.71:0/3583936401 wait complete. 2021-11-20T07:34:20.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c86c3e700 1 Processor -- start 2021-11-20T07:34:20.811 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c86c3e700 1 -- start start 2021-11-20T07:34:20.812 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c86c3e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:20.812 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c86c3e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8006fc70 con 0x7f4c800fa8a0 2021-11-20T07:34:20.812 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c85c3c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:20.812 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c85c3c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47174/0 (socket says 172.21.15.71:47174) 2021-11-20T07:34:20.813 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c85c3c700 1 -- 172.21.15.71:0/4255650135 learned_addr learned my addr 172.21.15.71:0/4255650135 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:20.814 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c85c3c700 1 -- 172.21.15.71:0/4255650135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c00b200 con 0x7f4c800fa8a0 2021-11-20T07:34:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c85c3c700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4c7c00e040 tx=0x7f4c7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.804+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7c01b410 con 0x7f4c800fa8a0 2021-11-20T07:34:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c7c01ba20 con 0x7f4c800fa8a0 2021-11-20T07:34:20.815 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8006fe70 con 0x7f4c800fa8a0 2021-11-20T07:34:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7c01a910 con 0x7f4c800fa8a0 2021-11-20T07:34:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8010db80 con 0x7f4c800fa8a0 2021-11-20T07:34:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4c7c00f070 con 0x7f4c800fa8a0 2021-11-20T07:34:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c6e7fc700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 0x7f4c68032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:20.817 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.805+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c80045ad0 con 0x7f4c800fa8a0 2021-11-20T07:34:20.817 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.806+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c7c041340 con 0x7f4c800fa8a0 2021-11-20T07:34:20.817 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.806+0000 7f4c8543b700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 0x7f4c68032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:20.817 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.806+0000 7f4c8543b700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 0x7f4c68032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4c700096a0 tx=0x7f4c70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:20.818 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.810+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c7c016030 con 0x7f4c800fa8a0 2021-11-20T07:34:20.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.959+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4c800fc680 con 0x7f4c800fa8a0 2021-11-20T07:34:20.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.960+0000 7f4c6e7fc700 1 -- 172.21.15.71:0/4255650135 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4c7c010540 con 0x7f4c800fa8a0 2021-11-20T07:34:20.962 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 8443 2021-11-20T07:34:20.964 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 msgr2=0x7f4c68032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 0x7f4c68032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4c700096a0 tx=0x7f4c70006b40).stop 2021-11-20T07:34:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 msgr2=0x7f4c8010d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4c7c00e040 tx=0x7f4c7c006ee0).stop 2021-11-20T07:34:20.965 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 shutdown_connections 2021-11-20T07:34:20.966 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4c6802feb0 0x7f4c68032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:20.966 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 --2- 172.21.15.71:0/4255650135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c800fa8a0 0x7f4c8010d4d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:20.966 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 >> 172.21.15.71:0/4255650135 conn(0x7f4c800f60e0 msgr2=0x7f4c800f7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:20.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 shutdown_connections 2021-11-20T07:34:20.967 INFO:teuthology.orchestra.run.smithi071.stderr:/usr/bin/ceph: 2021-11-20T07:34:20.961+0000 7f4c86c3e700 1 -- 172.21.15.71:0/4255650135 wait complete. 2021-11-20T07:34:21.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:20 smithi071 conmon[33687]: audit 2021-11-20T07:34:19.607515+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:34:21.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:20 smithi071 conmon[33687]: mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:21.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:20 smithi071 conmon[33687]: audit 2021-11-20T07:34:19.825369+0000 mon.smithi071 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:21.281 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T07:34:21.289 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: disabled 2021-11-20T07:34:21.289 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T07:34:21.296 INFO:teuthology.orchestra.run.smithi071.stderr:systemctl: inactive 2021-11-20T07:34:21.297 INFO:teuthology.orchestra.run.smithi071.stderr:firewalld.service is not enabled 2021-11-20T07:34:21.297 INFO:teuthology.orchestra.run.smithi071.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-11-20T07:34:21.300 INFO:teuthology.orchestra.run.smithi071.stderr:Ceph Dashboard is now available at: 2021-11-20T07:34:21.300 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.300 INFO:teuthology.orchestra.run.smithi071.stderr: URL: https://smithi071.front.sepia.ceph.com:8443/ 2021-11-20T07:34:21.300 INFO:teuthology.orchestra.run.smithi071.stderr: User: admin 2021-11-20T07:34:21.301 INFO:teuthology.orchestra.run.smithi071.stderr: Password: s7p19ha38p 2021-11-20T07:34:21.301 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.301 INFO:teuthology.orchestra.run.smithi071.stderr:You can access the Ceph CLI with: 2021-11-20T07:34:21.301 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.301 INFO:teuthology.orchestra.run.smithi071.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:34:21.302 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.302 INFO:teuthology.orchestra.run.smithi071.stderr:Please consider enabling telemetry to help improve Ceph: 2021-11-20T07:34:21.302 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.302 INFO:teuthology.orchestra.run.smithi071.stderr: ceph telemetry on 2021-11-20T07:34:21.303 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.303 INFO:teuthology.orchestra.run.smithi071.stderr:For more information see: 2021-11-20T07:34:21.303 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.303 INFO:teuthology.orchestra.run.smithi071.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-11-20T07:34:21.303 INFO:teuthology.orchestra.run.smithi071.stderr: 2021-11-20T07:34:21.304 INFO:teuthology.orchestra.run.smithi071.stderr:Bootstrap complete. 2021-11-20T07:34:21.304 INFO:teuthology.orchestra.run.smithi071.stderr:Releasing lock 139769234262504 on /run/cephadm/16910ea2-49d4-11ec-8c2c-001a4aab830c.lock 2021-11-20T07:34:21.304 INFO:teuthology.orchestra.run.smithi071.stderr:Lock 139769234262504 released on /run/cephadm/16910ea2-49d4-11ec-8c2c-001a4aab830c.lock 2021-11-20T07:34:21.336 INFO:tasks.cephadm:Fetching config... 2021-11-20T07:34:21.338 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:34:21.338 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-11-20T07:34:21.393 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-11-20T07:34:21.394 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:34:21.394 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-11-20T07:34:21.448 INFO:tasks.cephadm:Fetching mon keyring... 2021-11-20T07:34:21.449 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:34:21.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/keyring of=/dev/stdout 2021-11-20T07:34:21.516 INFO:tasks.cephadm:Fetching pub ssh key... 2021-11-20T07:34:21.517 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:34:21.517 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-11-20T07:34:21.574 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-11-20T07:34:21.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMgwV8eM3+Exm91hbL/V6gPvDHkDXVH3CSENx+9UHu8GzR8q+A3TXqVyWUbkd0Odc6lWcTXtZQ25hrEnDxk2S697H7R7iAjjnNQgDo5IsKt6GDx+1UllY69niTytQK6GfKBrlKnAYH8PKYdUtKwT5qP7T4by4le03YcRmqaIGiDmP1pb9F7clXAucCsafpvgDaZWYy2t101w27ryhFsUGSZvthhziE4/pLlJKEDc41z6zMpDuv2d9UIOrcTD4ykS0c8Ceh3xlrRGJfM6yxiwH+b0kOsBu3fsjQ6mwlJheCJb1Cfyim0z0FCV8prb+cOjOtRLZ0yBJTegxcE24i9EwBJMYh/aKeio1veYK4GhL6FvxgaEiEUSLq0brpKSQ7X0a+1DmUchYZafaLdaV5pr1FYndEVkMNEgJ1zD8Li/BKNr1QPBYgzSZbKdZZSBX7boDlvnFQgAk8aP2Hi7DvwsTZsvG/bUmwsglTZsCRMVQt+pSprOUfDYXgaEUg5YIoPP8= ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T07:34:21.651 INFO:teuthology.orchestra.run.smithi071.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMgwV8eM3+Exm91hbL/V6gPvDHkDXVH3CSENx+9UHu8GzR8q+A3TXqVyWUbkd0Odc6lWcTXtZQ25hrEnDxk2S697H7R7iAjjnNQgDo5IsKt6GDx+1UllY69niTytQK6GfKBrlKnAYH8PKYdUtKwT5qP7T4by4le03YcRmqaIGiDmP1pb9F7clXAucCsafpvgDaZWYy2t101w27ryhFsUGSZvthhziE4/pLlJKEDc41z6zMpDuv2d9UIOrcTD4ykS0c8Ceh3xlrRGJfM6yxiwH+b0kOsBu3fsjQ6mwlJheCJb1Cfyim0z0FCV8prb+cOjOtRLZ0yBJTegxcE24i9EwBJMYh/aKeio1veYK4GhL6FvxgaEiEUSLq0brpKSQ7X0a+1DmUchYZafaLdaV5pr1FYndEVkMNEgJ1zD8Li/BKNr1QPBYgzSZbKdZZSBX7boDlvnFQgAk8aP2Hi7DvwsTZsvG/bUmwsglTZsCRMVQt+pSprOUfDYXgaEUg5YIoPP8= ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:34:21.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMgwV8eM3+Exm91hbL/V6gPvDHkDXVH3CSENx+9UHu8GzR8q+A3TXqVyWUbkd0Odc6lWcTXtZQ25hrEnDxk2S697H7R7iAjjnNQgDo5IsKt6GDx+1UllY69niTytQK6GfKBrlKnAYH8PKYdUtKwT5qP7T4by4le03YcRmqaIGiDmP1pb9F7clXAucCsafpvgDaZWYy2t101w27ryhFsUGSZvthhziE4/pLlJKEDc41z6zMpDuv2d9UIOrcTD4ykS0c8Ceh3xlrRGJfM6yxiwH+b0kOsBu3fsjQ6mwlJheCJb1Cfyim0z0FCV8prb+cOjOtRLZ0yBJTegxcE24i9EwBJMYh/aKeio1veYK4GhL6FvxgaEiEUSLq0brpKSQ7X0a+1DmUchYZafaLdaV5pr1FYndEVkMNEgJ1zD8Li/BKNr1QPBYgzSZbKdZZSBX7boDlvnFQgAk8aP2Hi7DvwsTZsvG/bUmwsglTZsCRMVQt+pSprOUfDYXgaEUg5YIoPP8= ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T07:34:21.704 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMgwV8eM3+Exm91hbL/V6gPvDHkDXVH3CSENx+9UHu8GzR8q+A3TXqVyWUbkd0Odc6lWcTXtZQ25hrEnDxk2S697H7R7iAjjnNQgDo5IsKt6GDx+1UllY69niTytQK6GfKBrlKnAYH8PKYdUtKwT5qP7T4by4le03YcRmqaIGiDmP1pb9F7clXAucCsafpvgDaZWYy2t101w27ryhFsUGSZvthhziE4/pLlJKEDc41z6zMpDuv2d9UIOrcTD4ykS0c8Ceh3xlrRGJfM6yxiwH+b0kOsBu3fsjQ6mwlJheCJb1Cfyim0z0FCV8prb+cOjOtRLZ0yBJTegxcE24i9EwBJMYh/aKeio1veYK4GhL6FvxgaEiEUSLq0brpKSQ7X0a+1DmUchYZafaLdaV5pr1FYndEVkMNEgJ1zD8Li/BKNr1QPBYgzSZbKdZZSBX7boDlvnFQgAk8aP2Hi7DvwsTZsvG/bUmwsglTZsCRMVQt+pSprOUfDYXgaEUg5YIoPP8= ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:34:21.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-11-20T07:34:22.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:21 smithi071 conmon[33687]: audit 2021-11-20T07:34:20.961067+0000 mon.smithi071 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.71:0/4255650135' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-11-20T07:34:23.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- 172.21.15.71:0/3710348788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 msgr2=0x7f63c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:23.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 --2- 172.21.15.71:0/3710348788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c00f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f63b0009fb0 tx=0x7f63b0009b30).stop 2021-11-20T07:34:23.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- 172.21.15.71:0/3710348788 shutdown_connections 2021-11-20T07:34:23.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 --2- 172.21.15.71:0/3710348788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c00f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:23.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- 172.21.15.71:0/3710348788 >> 172.21.15.71:0/3710348788 conn(0x7f63c00f09a0 msgr2=0x7f63c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:23.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- 172.21.15.71:0/3710348788 shutdown_connections 2021-11-20T07:34:23.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- 172.21.15.71:0/3710348788 wait complete. 2021-11-20T07:34:23.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 Processor -- start 2021-11-20T07:34:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 -- start start 2021-11-20T07:34:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.481+0000 7f63bffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63bffff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c01069b0 con 0x7f63c00f51a0 2021-11-20T07:34:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63beffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63beffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47184/0 (socket says 172.21.15.71:47184) 2021-11-20T07:34:23.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63beffd700 1 -- 172.21.15.71:0/2412598632 learned_addr learned my addr 172.21.15.71:0/2412598632 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:23.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63beffd700 1 -- 172.21.15.71:0/2412598632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0005040 con 0x7f63c00f51a0 2021-11-20T07:34:23.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63beffd700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f63b0013510 tx=0x7f63b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:23.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b0009320 con 0x7f63c00f51a0 2021-11-20T07:34:23.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b0013f00 con 0x7f63c00f51a0 2021-11-20T07:34:23.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c0106bb0 con 0x7f63c00f51a0 2021-11-20T07:34:23.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b0014dc0 con 0x7f63c00f51a0 2021-11-20T07:34:23.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.482+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c0107040 con 0x7f63c00f51a0 2021-11-20T07:34:23.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63b001d3d0 con 0x7f63c00f51a0 2021-11-20T07:34:23.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c0045ad0 con 0x7f63c00f51a0 2021-11-20T07:34:23.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63a77fe700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 0x7f63a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:23.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63b0041040 con 0x7f63c00f51a0 2021-11-20T07:34:23.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63be7fc700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 0x7f63a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:23.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.483+0000 7f63be7fc700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 0x7f63a8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:23.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.487+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63b00126f0 con 0x7f63c00f51a0 2021-11-20T07:34:23.644 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.642+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f63c00fabe0 con 0x7f63c00f51a0 2021-11-20T07:34:23.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.645+0000 7f63a77fe700 1 -- 172.21.15.71:0/2412598632 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f63b0012330 con 0x7f63c00f51a0 2021-11-20T07:34:23.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 msgr2=0x7f63a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:23.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 0x7f63a8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).stop 2021-11-20T07:34:23.650 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 msgr2=0x7f63c01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:23.650 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f63b0013510 tx=0x7f63b000b040).stop 2021-11-20T07:34:23.650 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 shutdown_connections 2021-11-20T07:34:23.650 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f63a802ff60 0x7f63a8032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:23.651 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 --2- 172.21.15.71:0/2412598632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63c00f51a0 0x7f63c01064c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:23.651 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 >> 172.21.15.71:0/2412598632 conn(0x7f63c00f09a0 msgr2=0x7f63c00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:23.651 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.647+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 shutdown_connections 2021-11-20T07:34:23.651 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:23.648+0000 7f63bffff700 1 -- 172.21.15.71:0/2412598632 wait complete. 2021-11-20T07:34:24.067 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:23 smithi071 conmon[33687]: cluster 2021-11-20T07:34:22.827818+0000 mon.smithi071 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi071.ynqmwn(active, since 6s) 2021-11-20T07:34:24.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:23 smithi071 conmon[33687]: audit 2021-11-20T07:34:23.646997+0000 mon.smithi071 (mon.0) 113 : audit [INF] from='client.? 172.21.15.71:0/2412598632' entity='client.admin' 2021-11-20T07:34:24.322 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2021-11-20T07:34:24.323 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:34:24.323 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2021-11-20T07:34:24.342 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:34:24.343 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T07:34:24.400 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2021-11-20T07:34:24.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch host add smithi112 2021-11-20T07:34:28.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:27 smithi071 conmon[33687]: audit 2021-11-20T07:34:26.805520+0000 mon.smithi071 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:28.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:27 smithi071 conmon[33687]: audit 2021-11-20T07:34:27.181239+0000 mon.smithi071 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:28.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:27 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:27.187591+0000 mgr.smithi071.ynqmwn (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi071 on smithi071 2021-11-20T07:34:33.659 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: audit 2021-11-20T07:34:32.387428+0000 mon.smithi071 (mon.0 2021-11-20T07:34:33.659 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: ) 116 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:33.659 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: audit 2021-11-20T07:34:32.388440+0000 mon.smithi071 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi071", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:34:33.659 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: audit 2021-11-20T07:34:32.390039+0000 mon.smithi071 (mon.0) 118 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi071", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T07:34:33.660 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: audit 2021-11-20T07:34:32.390651+0000 mon.smithi071 (mon.0) 119 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:34:33.660 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:33 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:32.391174+0000 mgr.smithi071.ynqmwn (mgr.14162) 6 : cephadm [INF] Deploying daemon crash.smithi071 on smithi071 2021-11-20T07:34:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.346161+0000 mon.smithi071 (mon. 2021-11-20T07:34:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: 0) 120 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.429693+0000 mon.smithi071 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.431569+0000 mon.smithi071 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.431935+0000 mon.smithi071 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T07:34:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:35 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.435391+0000 mon.smithi071 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:36.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:36 smithi071 conmon[33687]: audit 2021-11-20T07:34:34.432346+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:34:36.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:36 smithi071 conmon[33687]: ) 7 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T07:34:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:36 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:34.438581+0000 mgr.smithi071.ynqmwn (mgr.14162) 8 : cephadm [INF] Deploying daemon grafana.smithi071 on smithi071 2021-11-20T07:34:37.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:37 smithi071 conmon[33687]: cluster 2021-11-20T07:34:36.324768+0000 mgr.smithi071.ynqmwn (mgr.14162) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:39.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:39 smithi071 conmon[33687]: cluster 2021-11-20T07:34:38.325027+0000 mgr.smithi071.ynqmwn (mgr.14162) 10 : cluster [DBG] 2021-11-20T07:34:39.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:39 smithi071 conmon[33687]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:39.888 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.880+0000 7f620d6cb700 1 -- 172.21.15.112:0/1020840741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 msgr2=0x7f62080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:39.889 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.880+0000 7f620d6cb700 1 --2- 172.21.15.112:0/1020840741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f62080f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f61f8009fb0 tx=0x7f61f8009b30).stop 2021-11-20T07:34:39.889 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 -- 172.21.15.112:0/1020840741 shutdown_connections 2021-11-20T07:34:39.890 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 --2- 172.21.15.112:0/1020840741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f62080f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:39.890 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 -- 172.21.15.112:0/1020840741 >> 172.21.15.112:0/1020840741 conn(0x7f62080f09a0 msgr2=0x7f62080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:39.891 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 -- 172.21.15.112:0/1020840741 shutdown_connections 2021-11-20T07:34:39.891 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 -- 172.21.15.112:0/1020840741 wait complete. 2021-11-20T07:34:39.892 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 Processor -- start 2021-11-20T07:34:39.892 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 -- start start 2021-11-20T07:34:39.893 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.881+0000 7f620d6cb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:39.893 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f620d6cb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6208060280 con 0x7f62080f51a0 2021-11-20T07:34:39.893 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6207fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:39.894 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6207fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43582/0 (socket says 172.21.15.112:43582) 2021-11-20T07:34:39.894 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6207fff700 1 -- 172.21.15.112:0/199048927 learned_addr learned my addr 172.21.15.112:0/199048927 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:34:39.895 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6207fff700 1 -- 172.21.15.112:0/199048927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8005040 con 0x7f62080f51a0 2021-11-20T07:34:39.895 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6207fff700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f61f8013510 tx=0x7f61f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:39.896 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f8009320 con 0x7f62080f51a0 2021-11-20T07:34:39.896 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6208060480 con 0x7f62080f51a0 2021-11-20T07:34:39.897 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61f8013f00 con 0x7f62080f51a0 2021-11-20T07:34:39.898 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f8014dc0 con 0x7f62080f51a0 2021-11-20T07:34:39.899 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.882+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62081098c0 con 0x7f62080f51a0 2021-11-20T07:34:39.900 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.883+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61f8014420 con 0x7f62080f51a0 2021-11-20T07:34:39.900 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.883+0000 7f6204ff9700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 0x7f61f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:39.900 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.883+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6208045ad0 con 0x7f62080f51a0 2021-11-20T07:34:39.900 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.883+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61f8041050 con 0x7f62080f51a0 2021-11-20T07:34:39.901 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.884+0000 7f62077fe700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 0x7f61f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:39.901 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.884+0000 7f62077fe700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 0x7f61f0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f61fc0096a0 tx=0x7f61fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:39.901 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:39.892+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61f80095f0 con 0x7f62080f51a0 2021-11-20T07:34:40.064 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.063+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7f62080f1b30 con 0x7f61f002ff60 2021-11-20T07:34:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.752+0000 7f6204ff9700 1 -- 172.21.15.112:0/199048927 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f62080f1b30 con 0x7f61f002ff60 2021-11-20T07:34:40.754 INFO:teuthology.orchestra.run.smithi112.stdout:Added host 'smithi112' 2021-11-20T07:34:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 msgr2=0x7f61f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 0x7f61f0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f61fc0096a0 tx=0x7f61fc006b40).stop 2021-11-20T07:34:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 msgr2=0x7f6208109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f61f8013510 tx=0x7f61f800b040).stop 2021-11-20T07:34:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 shutdown_connections 2021-11-20T07:34:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f61f002ff60 0x7f61f0032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 --2- 172.21.15.112:0/199048927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62080f51a0 0x7f6208109190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 >> 172.21.15.112:0/199048927 conn(0x7f62080f09a0 msgr2=0x7f62080f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 shutdown_connections 2021-11-20T07:34:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:40.754+0000 7f620d6cb700 1 -- 172.21.15.112:0/199048927 wait complete. 2021-11-20T07:34:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:41 smithi071 conmon[33687]: audit 2021-11-20T07:34:40.064420+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:34:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:41 smithi071 conmon[33687]: ) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:41 smithi071 conmon[33687]: cluster 2021-11-20T07:34:40.325334+0000 mgr.smithi071.ynqmwn (mgr.14162) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:41 smithi071 conmon[33687]: audit 2021-11-20T07:34:40. 2021-11-20T07:34:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:41 smithi071 conmon[33687]: 752011+0000 mon.smithi071 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:41.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch host ls --format=json 2021-11-20T07:34:42.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:42 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:40.752325+0000 2021-11-20T07:34:42.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:42 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 13 : cephadm [INF] Added host smithi112 2021-11-20T07:34:43.587 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:43 smithi071 conmon[33687]: cluster 2021-11-20T07:34:42.325695+0000 mgr.smithi071.ynqmwn (mgr.14162) 14 2021-11-20T07:34:43.587 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:43 smithi071 conmon[33687]: : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:44.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:44 smithi071 conmon[33687]: cluster 2021-11-20T07:34:44.326070+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:34:44.447 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:44 smithi071 conmon[33687]: 14162) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:45.913 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:45 smithi071 conmon[33687]: audit 2021-11-20T07:34:44.761493+0000 mon.smithi071 2021-11-20T07:34:45.914 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:45 smithi071 conmon[33687]: (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:45.914 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:45 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:44.762799+0000 mgr.smithi071.ynqmwn (mgr.14162) 16 : cephadm [INF] Deploying daemon node-exporter.smithi071 on smithi071 2021-11-20T07:34:47.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:46 smithi071 conmon[33687]: cluster 2021-11-20T07:34:46.326481+0000 mgr.smithi071.ynqmwn (mgr.14162) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:47.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:46 smithi071 conmon[33687]: 2021-11-20T07:34:47.420 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1285090793 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 msgr2=0x7fc0a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:47.420 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1285090793 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a80f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc098009fb0 tx=0x7fc098009b30).stop 2021-11-20T07:34:47.420 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1285090793 shutdown_connections 2021-11-20T07:34:47.421 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1285090793 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a80f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:47.421 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1285090793 >> 172.21.15.112:0/1285090793 conn(0x7fc0a80f0b70 msgr2=0x7fc0a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:47.421 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1285090793 shutdown_connections 2021-11-20T07:34:47.421 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.227+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1285090793 wait complete. 2021-11-20T07:34:47.422 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0acfe0700 1 Processor -- start 2021-11-20T07:34:47.422 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0acfe0700 1 -- start start 2021-11-20T07:34:47.422 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0acfe0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:47.422 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0acfe0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8106bd0 con 0x7fc0a80f5410 2021-11-20T07:34:47.422 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0a659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:47.423 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0a659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43588/0 (socket says 172.21.15.112:43588) 2021-11-20T07:34:47.423 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0a659c700 1 -- 172.21.15.112:0/1751313021 learned_addr learned my addr 172.21.15.112:0/1751313021 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:34:47.423 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0a659c700 1 -- 172.21.15.112:0/1751313021 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098005040 con 0x7fc0a80f5410 2021-11-20T07:34:47.423 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.228+0000 7fc0a659c700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc098013510 tx=0x7fc09800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:47.424 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc098009320 con 0x7fc0a80f5410 2021-11-20T07:34:47.424 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc098013f00 con 0x7fc0a80f5410 2021-11-20T07:34:47.425 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc098014d90 con 0x7fc0a80f5410 2021-11-20T07:34:47.425 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a8106dd0 con 0x7fc0a80f5410 2021-11-20T07:34:47.425 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8107260 con 0x7fc0a80f5410 2021-11-20T07:34:47.425 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc098014420 con 0x7fc0a80f5410 2021-11-20T07:34:47.425 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a8045ad0 con 0x7fc0a80f5410 2021-11-20T07:34:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.229+0000 7fc09effd700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 0x7fc094032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.230+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc098041060 con 0x7fc0a80f5410 2021-11-20T07:34:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.230+0000 7fc09ffff700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 0x7fc094032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.230+0000 7fc09ffff700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 0x7fc094032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:47.427 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:34:47.427 INFO:teuthology.orchestra.run.smithi112.stdout:[{"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}, {"addr": "smithi112", "hostname": "smithi112", "labels": [], "status": ""}] 2021-11-20T07:34:47.428 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.234+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc09806d2b0 con 0x7fc0a80f5410 2021-11-20T07:34:47.428 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.403+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0a80f29d0 con 0x7fc09402ffb0 2021-11-20T07:34:47.428 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.405+0000 7fc09effd700 1 -- 172.21.15.112:0/1751313021 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fc0a80f29d0 con 0x7fc09402ffb0 2021-11-20T07:34:47.428 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 msgr2=0x7fc094032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 0x7fc094032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).stop 2021-11-20T07:34:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 msgr2=0x7fc0a81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc098013510 tx=0x7fc09800b040).stop 2021-11-20T07:34:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 shutdown_connections 2021-11-20T07:34:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc09402ffb0 0x7fc094032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 --2- 172.21.15.112:0/1751313021 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0a80f5410 0x7fc0a81066e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.406+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 >> 172.21.15.112:0/1751313021 conn(0x7fc0a80f0b70 msgr2=0x7fc0a805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.407+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 shutdown_connections 2021-11-20T07:34:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:47.407+0000 7fc0acfe0700 1 -- 172.21.15.112:0/1751313021 wait complete. 2021-11-20T07:34:48.251 INFO:tasks.cephadm:Setting crush tunables to default 2021-11-20T07:34:48.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd crush tunables default 2021-11-20T07:34:48.981 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:48 smithi071 conmon[33687]: audit 2021-11-20T07:34:47.406115+0000 mgr.smithi071.ynqmwn (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T07:34:48.982 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:48 smithi071 conmon[33687]: audit 2021-11-20T07:34:47.804982+0000 mon.smithi071 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:48.982 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:48 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:47.810225+0000 mgr.smithi071.ynqmwn (mgr.14162) 19 : cephadm [INF] Deploying daemon prometheus.smithi071 on smithi071 2021-11-20T07:34:48.982 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:48 smithi071 conmon[33687]: cluster 2021-11-20T07:34:48.326713+0000 mgr.smithi071.ynqmwn (mgr.14162) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:50.682 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- 172.21.15.71:0/74000646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 msgr2=0x7f99340f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:50.682 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 --2- 172.21.15.71:0/74000646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f99340f4f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9928009fb0 tx=0x7f9928009b30).stop 2021-11-20T07:34:50.683 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- 172.21.15.71:0/74000646 shutdown_connections 2021-11-20T07:34:50.683 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 --2- 172.21.15.71:0/74000646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f99340f4f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:50.683 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- 172.21.15.71:0/74000646 >> 172.21.15.71:0/74000646 conn(0x7f99340f0260 msgr2=0x7f99340f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:50.683 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- 172.21.15.71:0/74000646 shutdown_connections 2021-11-20T07:34:50.684 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- 172.21.15.71:0/74000646 wait complete. 2021-11-20T07:34:50.684 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 Processor -- start 2021-11-20T07:34:50.684 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.680+0000 7f993b055700 1 -- start start 2021-11-20T07:34:50.684 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f993b055700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f993b055700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993410d220 con 0x7f99340f4b00 2021-11-20T07:34:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f9938df1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f9938df1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47268/0 (socket says 172.21.15.71:47268) 2021-11-20T07:34:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f9938df1700 1 -- 172.21.15.71:0/877168729 learned_addr learned my addr 172.21.15.71:0/877168729 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:34:50.685 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f9938df1700 1 -- 172.21.15.71:0/877168729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928005040 con 0x7f99340f4b00 2021-11-20T07:34:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f9938df1700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9928005010 tx=0x7f992800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99280138c0 con 0x7f99340f4b00 2021-11-20T07:34:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9928014030 con 0x7f99340f4b00 2021-11-20T07:34:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.681+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f992801c5c0 con 0x7f99340f4b00 2021-11-20T07:34:50.686 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.682+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993410d420 con 0x7f99340f4b00 2021-11-20T07:34:50.687 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.682+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993410e030 con 0x7f99340f4b00 2021-11-20T07:34:50.687 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9928024070 con 0x7f99340f4b00 2021-11-20T07:34:50.688 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993410d8b0 con 0x7f99340f4b00 2021-11-20T07:34:50.688 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f99317fa700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 0x7f9924032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:50.688 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9928040bc0 con 0x7f99340f4b00 2021-11-20T07:34:50.689 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f9933fff700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 0x7f9924032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:50.689 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.683+0000 7f9933fff700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 0x7f9924032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:50.690 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.688+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f993410d8b0 con 0x7f99340f4b00 2021-11-20T07:34:50.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:50.846+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f993410d8b0 con 0x7f99340f4b00 2021-11-20T07:34:51.320 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.319+0000 7f99317fa700 1 -- 172.21.15.71:0/877168729 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f993410d8b0 con 0x7f99340f4b00 2021-11-20T07:34:51.325 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 msgr2=0x7f9924032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:51.326 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 0x7f9924032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).stop 2021-11-20T07:34:51.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 msgr2=0x7f993410cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:51.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9928005010 tx=0x7f992800b040).stop 2021-11-20T07:34:51.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 shutdown_connections 2021-11-20T07:34:51.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f992402ffb0 0x7f9924032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:51.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 --2- 172.21.15.71:0/877168729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99340f4b00 0x7f993410cd30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 >> 172.21.15.71:0/877168729 conn(0x7f99340f0260 msgr2=0x7f99340fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:51.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 shutdown_connections 2021-11-20T07:34:51.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:34:51.321+0000 7f993b055700 1 -- 172.21.15.71:0/877168729 wait complete. 2021-11-20T07:34:51.332 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2021-11-20T07:34:51.817 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:51 smithi071 conmon[33687]: cluster 2021-11-20T07:34:50.326991+0000 mgr.smithi071.ynqmwn (mgr.14162) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:51 smithi071 conmon[33687]: audit 2021-11-20T07:34:50.847345+0000 mon.smithi071 (mon.0) 128 : audit [INF] from='client.? 172.21.15.71:0/877168729' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-11-20T07:34:52.275 INFO:tasks.cephadm:Adding mon.smithi071 on smithi071 2021-11-20T07:34:52.276 INFO:tasks.cephadm:Adding mon.smithi112 on smithi112 2021-11-20T07:34:52.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch apply mon '2;smithi071:172.21.15.71=smithi071;smithi112:172.21.15.112=smithi112' 2021-11-20T07:34:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:52 smithi071 conmon[33687]: audit 2021-11-20T07:34:51.319097+0000 mon.smithi071 ( 2021-11-20T07:34:52.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:52 smithi071 conmon[33687]: mon.0) 129 : audit [INF] from='client.? 172.21.15.71:0/877168729' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-11-20T07:34:52.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:52 smithi071 conmon[33687]: cluster 2021-11-20T07:34:51.319168+0000 mon.smithi071 (mon.0) 130 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T07:34:53.628 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:53 smithi071 conmon[33687]: cluster 2021-11-20T07:34:52.327235+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:34:53.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:53 smithi071 conmon[33687]: 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:54.003 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- 172.21.15.112:0/1812365001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 msgr2=0x7f35040f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:54.004 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 --2- 172.21.15.112:0/1812365001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f35040f5750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3500009fb0 tx=0x7f3500009b30).stop 2021-11-20T07:34:54.004 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- 172.21.15.112:0/1812365001 shutdown_connections 2021-11-20T07:34:54.004 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 --2- 172.21.15.112:0/1812365001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f35040f5750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:54.004 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- 172.21.15.112:0/1812365001 >> 172.21.15.112:0/1812365001 conn(0x7f35040f0a90 msgr2=0x7f35040f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:54.004 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- 172.21.15.112:0/1812365001 shutdown_connections 2021-11-20T07:34:54.005 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- 172.21.15.112:0/1812365001 wait complete. 2021-11-20T07:34:54.005 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 Processor -- start 2021-11-20T07:34:54.005 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:53.999+0000 7f350c3e5700 1 -- start start 2021-11-20T07:34:54.005 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350c3e5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:54.006 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350c3e5700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504107470 con 0x7f35040f5330 2021-11-20T07:34:54.006 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350a181700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:54.006 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350a181700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43594/0 (socket says 172.21.15.112:43594) 2021-11-20T07:34:54.008 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350a181700 1 -- 172.21.15.112:0/2537309668 learned_addr learned my addr 172.21.15.112:0/2537309668 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:34:54.009 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350a181700 1 -- 172.21.15.112:0/2537309668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500005040 con 0x7f35040f5330 2021-11-20T07:34:54.009 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f350a181700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3500014000 tx=0x7f350000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:54.009 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35000095f0 con 0x7f35040f5330 2021-11-20T07:34:54.009 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.000+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3500007e70 con 0x7f35040f5330 2021-11-20T07:34:54.010 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3504060220 con 0x7f35040f5330 2021-11-20T07:34:54.010 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3500014ea0 con 0x7f35040f5330 2021-11-20T07:34:54.010 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350410bf70 con 0x7f35040f5330 2021-11-20T07:34:54.010 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3500025070 con 0x7f35040f5330 2021-11-20T07:34:54.011 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3504045ad0 con 0x7f35040f5330 2021-11-20T07:34:54.011 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f34faffd700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 0x7f34f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:54.011 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.001+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35000409f0 con 0x7f35040f5330 2021-11-20T07:34:54.011 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.002+0000 7f3509980700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 0x7f34f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:54.011 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.002+0000 7f3509980700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 0x7f34f0032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f34f40096a0 tx=0x7f34f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:54.012 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.006+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f350000f4e0 con 0x7f35040f5330 2021-11-20T07:34:54.183 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.181+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi071:172.21.15.71=smithi071;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7f35040f1cf0 con 0x7f34f002ff60 2021-11-20T07:34:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.186+0000 7f34faffd700 1 -- 172.21.15.112:0/2537309668 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35040f1cf0 con 0x7f34f002ff60 2021-11-20T07:34:54.188 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mon update... 2021-11-20T07:34:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 msgr2=0x7f34f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 0x7f34f0032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f34f40096a0 tx=0x7f34f4006b40).stop 2021-11-20T07:34:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 msgr2=0x7f350410ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3500014000 tx=0x7f350000b040).stop 2021-11-20T07:34:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 shutdown_connections 2021-11-20T07:34:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f34f002ff60 0x7f34f0032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 --2- 172.21.15.112:0/2537309668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35040f5330 0x7f350410ba80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 >> 172.21.15.112:0/2537309668 conn(0x7f35040f0a90 msgr2=0x7f35040f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 shutdown_connections 2021-11-20T07:34:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:54.187+0000 7f350c3e5700 1 -- 172.21.15.112:0/2537309668 wait complete. 2021-11-20T07:34:54.738 DEBUG:teuthology.orchestra.run.smithi112:mon.smithi112> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi112.service 2021-11-20T07:34:54.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T07:34:54.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T07:34:54.768 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:-- Logs begin at Sat 2021-11-20 07:22:39 UTC. -- 2021-11-20T07:34:55.484 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:55 smithi071 conmon[33687]: audit 2021-11-20T07:34:54.183780+0000 mgr.smithi071.ynqmwn (mgr.14162) 23 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi071:172.21.15.71=smithi071;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:34:55.484 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:55 smithi071 conmon[33687]: cephadm 2021-11-20T07:34:54.184940+0000 mgr.smithi071.ynqmwn (mgr.14162) 24 : cephadm [INF] Saving service mon spec with placement smithi071:172.21.15.71=smithi071;smithi112:172.21.15.112=smithi112;count:2 2021-11-20T07:34:55.484 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:55 smithi071 conmon[33687]: audit 2021-11-20T07:34:54.187328+0000 mon.smithi071 (mon.0) 131 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:55.484 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:55 smithi071 conmon[33687]: cluster 2021-11-20T07:34:54.327445+0000 mgr.smithi071.ynqmwn (mgr.14162) 25 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:56.398 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 -- 172.21.15.112:0/2668871655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 msgr2=0x7ff4fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:56.398 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 --2- 172.21.15.112:0/2668871655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc0f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff4f8009fb0 tx=0x7ff4f8009b30).stop 2021-11-20T07:34:56.398 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 -- 172.21.15.112:0/2668871655 shutdown_connections 2021-11-20T07:34:56.398 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 --2- 172.21.15.112:0/2668871655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc0f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:56.399 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 -- 172.21.15.112:0/2668871655 >> 172.21.15.112:0/2668871655 conn(0x7ff4fc0f1c70 msgr2=0x7ff4fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:56.399 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 -- 172.21.15.112:0/2668871655 shutdown_connections 2021-11-20T07:34:56.399 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.393+0000 7ff504524700 1 -- 172.21.15.112:0/2668871655 wait complete. 2021-11-20T07:34:56.399 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff504524700 1 Processor -- start 2021-11-20T07:34:56.400 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff504524700 1 -- start start 2021-11-20T07:34:56.400 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff504524700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:56.400 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff504524700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc106bc0 con 0x7ff4fc0f6510 2021-11-20T07:34:56.400 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff5022c0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff5022c0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43600/0 (socket says 172.21.15.112:43600) 2021-11-20T07:34:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff5022c0700 1 -- 172.21.15.112:0/3544390459 learned_addr learned my addr 172.21.15.112:0/3544390459 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:34:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff5022c0700 1 -- 172.21.15.112:0/3544390459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8005040 con 0x7ff4fc0f6510 2021-11-20T07:34:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff5022c0700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff4f8013510 tx=0x7ff4f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:56.401 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f8009320 con 0x7ff4fc0f6510 2021-11-20T07:34:56.402 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4f8013f00 con 0x7ff4fc0f6510 2021-11-20T07:34:56.402 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.394+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4fc106dc0 con 0x7ff4fc0f6510 2021-11-20T07:34:56.402 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f8014d90 con 0x7ff4fc0f6510 2021-11-20T07:34:56.402 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4fc107250 con 0x7ff4fc0f6510 2021-11-20T07:34:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4f8014420 con 0x7ff4fc0f6510 2021-11-20T07:34:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff4f2ffd700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 0x7ff4e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:34:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4f8041090 con 0x7ff4fc0f6510 2021-11-20T07:34:56.403 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.395+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4fc045ad0 con 0x7ff4fc0f6510 2021-11-20T07:34:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.396+0000 7ff501abf700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 0x7ff4e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:34:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.396+0000 7ff501abf700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 0x7ff4e8032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff4ec0096a0 tx=0x7ff4ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:34:56.404 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.400+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4f80095f0 con 0x7ff4fc0f6510 2021-11-20T07:34:56.618 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.616+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4fc108100 con 0x7ff4fc0f6510 2021-11-20T07:34:56.619 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.617+0000 7ff4f2ffd700 1 -- 172.21.15.112:0/3544390459 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4f801db70 con 0x7ff4fc0f6510 2021-11-20T07:34:56.619 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:34:56.620 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","modified":"2021-11-20T07:33:19.655860Z","created":"2021-11-20T07:33:19.655860Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi071","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T07:34:56.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 msgr2=0x7ff4e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:56.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 0x7ff4e8032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff4ec0096a0 tx=0x7ff4ec006b40).stop 2021-11-20T07:34:56.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 msgr2=0x7ff4fc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:34:56.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff4f8013510 tx=0x7ff4f800b040).stop 2021-11-20T07:34:56.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 shutdown_connections 2021-11-20T07:34:56.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff4e802ffb0 0x7ff4e8032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:56.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 --2- 172.21.15.112:0/3544390459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4fc0f6510 0x7ff4fc1066d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:34:56.623 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 >> 172.21.15.112:0/3544390459 conn(0x7ff4fc0f1c70 msgr2=0x7ff4fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:34:56.623 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 shutdown_connections 2021-11-20T07:34:56.623 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:34:56.619+0000 7ff504524700 1 -- 172.21.15.112:0/3544390459 wait complete. 2021-11-20T07:34:56.623 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2021-11-20T07:34:57.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: audit 2021-11-20T07:34:55.832403+0000 mon.smithi071 (mon.0) 132 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:57.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: audit 2021-11-20T07:34:55.832941+0000 mon.smithi071 (mon.0) 133 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:34:57.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: audit 2021-11-20T07:34:56.196738+0000 mon.smithi071 (mon.0) 134 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:57.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: cluster 2021-11-20T07:34:56.327725+0000 mgr.smithi071.ynqmwn (mgr.14162) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:34:57.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: audit 2021-11-20T07:34:56.618746+0000 mon.smithi071 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.112:0/3544390459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:34:57.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:56 smithi071 conmon[33687]: audit 2021-11-20T07:34:56.636560+0000 mon.smithi071 (mon.0) 136 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:34:58.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T07:34:58.119 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T07:34:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:58 smithi071 conmon[33687]: cluster 2021-11-20T07:34:58.328000+0000 mgr.smithi071.ynqmwn (mgr.14162) 27 2021-11-20T07:34:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:34:58 smithi071 conmon[33687]: : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:00.614 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- 172.21.15.112:0/2799358014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 msgr2=0x7f4a980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:00.614 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 --2- 172.21.15.112:0/2799358014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a980f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f4a94009fb0 tx=0x7f4a94009b30).stop 2021-11-20T07:35:00.614 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- 172.21.15.112:0/2799358014 shutdown_connections 2021-11-20T07:35:00.615 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 --2- 172.21.15.112:0/2799358014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a980f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:00.615 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- 172.21.15.112:0/2799358014 >> 172.21.15.112:0/2799358014 conn(0x7f4a980f0b70 msgr2=0x7f4a980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:00.616 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- 172.21.15.112:0/2799358014 shutdown_connections 2021-11-20T07:35:00.616 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- 172.21.15.112:0/2799358014 wait complete. 2021-11-20T07:35:00.616 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 Processor -- start 2021-11-20T07:35:00.617 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.610+0000 7f4a9f766700 1 -- start start 2021-11-20T07:35:00.617 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9f766700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:00.617 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9f766700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98106df0 con 0x7f4a980f5410 2021-11-20T07:35:00.617 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9d502700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:00.617 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9d502700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43606/0 (socket says 172.21.15.112:43606) 2021-11-20T07:35:00.618 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9d502700 1 -- 172.21.15.112:0/3857009038 learned_addr learned my addr 172.21.15.112:0/3857009038 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:35:00.618 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9d502700 1 -- 172.21.15.112:0/3857009038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94005040 con 0x7f4a980f5410 2021-11-20T07:35:00.618 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.611+0000 7f4a9d502700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a94013510 tx=0x7f4a9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:00.619 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.612+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a94009320 con 0x7f4a980f5410 2021-11-20T07:35:00.619 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.612+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9810b8f0 con 0x7f4a980f5410 2021-11-20T07:35:00.619 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.612+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a94013f00 con 0x7f4a980f5410 2021-11-20T07:35:00.619 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.612+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a94014d90 con 0x7f4a980f5410 2021-11-20T07:35:00.620 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.612+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9810bcc0 con 0x7f4a980f5410 2021-11-20T07:35:00.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a940263a0 con 0x7f4a980f5410 2021-11-20T07:35:00.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a8dffb700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 0x7f4a84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:00.621 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a94041f30 con 0x7f4a980f5410 2021-11-20T07:35:00.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98059100 con 0x7f4a980f5410 2021-11-20T07:35:00.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a9cd01700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 0x7f4a84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:00.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.613+0000 7f4a9cd01700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 0x7f4a84032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4a880096a0 tx=0x7f4a88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:00.622 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.617+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a94012920 con 0x7f4a980f5410 2021-11-20T07:35:00.852 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.850+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a98045ad0 con 0x7f4a980f5410 2021-11-20T07:35:00.853 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.851+0000 7f4a8dffb700 1 -- 172.21.15.112:0/3857009038 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a9402b020 con 0x7f4a980f5410 2021-11-20T07:35:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:35:00.854 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","modified":"2021-11-20T07:33:19.655860Z","created":"2021-11-20T07:33:19.655860Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi071","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T07:35:00.855 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 msgr2=0x7f4a84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:00.856 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 0x7f4a84032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4a880096a0 tx=0x7f4a88006b40).stop 2021-11-20T07:35:00.857 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 msgr2=0x7f4a9810b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:00.857 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a94013510 tx=0x7f4a9400b040).stop 2021-11-20T07:35:00.858 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 shutdown_connections 2021-11-20T07:35:00.859 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4a8402ffb0 0x7f4a84032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:00.859 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 --2- 172.21.15.112:0/3857009038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a980f5410 0x7f4a9810b400 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:00.860 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 >> 172.21.15.112:0/3857009038 conn(0x7f4a980f0b70 msgr2=0x7f4a980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:00.860 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 shutdown_connections 2021-11-20T07:35:00.861 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:00.853+0000 7f4a9f766700 1 -- 172.21.15.112:0/3857009038 wait complete. 2021-11-20T07:35:00.861 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2021-11-20T07:35:01.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35:00.099858+0000 mon.smithi071 (mon.0) 137 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:01.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:35:01.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: 35:00.328242+0000 mgr.smithi071.ynqmwn (mgr.14162) 28 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:01.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35:00.427338+0000 mon.smithi071 (mon.0) 138 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:01.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35:00.805361+0000 mon.smithi071 (mon. 2021-11-20T07:35:01.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: 0) 139 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:01.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35:00.807874+0000 mon.smithi071 2021-11-20T07:35:01.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:01.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:01.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: -20T07:35:00.808895+0000 mon.smithi071 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:35:01.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35: 2021-11-20T07:35:01.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: 00.813666+0000 mon.smithi071 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T07:35:01.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35 2021-11-20T07:35:01.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: :00.814391+0000 mon.smithi071 (mon.0) 143 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:01.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: audit 2021-11-20T07:35 2021-11-20T07:35:01.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:01 smithi071 conmon[33687]: :00.852663+0000 mon.smithi071 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.112:0/3857009038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:35:02.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T07:35:02.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T07:35:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:02 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:00.814988+0000 mgr.smithi071.ynqmwn (mgr.14162) 29 : cephadm 2021-11-20T07:35:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:02 smithi071 conmon[33687]: [INF] Deploying daemon crash.smithi112 on smithi112 2021-11-20T07:35:03.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:03 smithi071 conmon[33687]: cluster 2021-11-20T07:35:02.328559+0000 mgr.smithi071.ynqmwn (mgr.14162) 30 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:04.307 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 -- 172.21.15.112:0/127927354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 msgr2=0x7fecd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:04.307 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 --2- 172.21.15.112:0/127927354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd80f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fecd4009fb0 tx=0x7fecd4009b30).stop 2021-11-20T07:35:04.308 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 -- 172.21.15.112:0/127927354 shutdown_connections 2021-11-20T07:35:04.308 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 --2- 172.21.15.112:0/127927354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd80f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:04.308 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 -- 172.21.15.112:0/127927354 >> 172.21.15.112:0/127927354 conn(0x7fecd80f09a0 msgr2=0x7fecd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:04.308 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.303+0000 7fecde9ed700 1 -- 172.21.15.112:0/127927354 shutdown_connections 2021-11-20T07:35:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecde9ed700 1 -- 172.21.15.112:0/127927354 wait complete. 2021-11-20T07:35:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecde9ed700 1 Processor -- start 2021-11-20T07:35:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecde9ed700 1 -- start start 2021-11-20T07:35:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecde9ed700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:04.309 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecde9ed700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd81058a0 con 0x7fecd80f51a0 2021-11-20T07:35:04.310 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecdd9eb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:04.310 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecdd9eb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43612/0 (socket says 172.21.15.112:43612) 2021-11-20T07:35:04.310 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.304+0000 7fecdd9eb700 1 -- 172.21.15.112:0/1629298988 learned_addr learned my addr 172.21.15.112:0/1629298988 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:35:04.310 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecdd9eb700 1 -- 172.21.15.112:0/1629298988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4005040 con 0x7fecd80f51a0 2021-11-20T07:35:04.311 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecdd9eb700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fecd4009830 tx=0x7fecd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:04.311 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd4009320 con 0x7fecd80f51a0 2021-11-20T07:35:04.311 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd8105aa0 con 0x7fecd80f51a0 2021-11-20T07:35:04.311 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecd40136c0 con 0x7fecd80f51a0 2021-11-20T07:35:04.311 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd401ee90 con 0x7fecd80f51a0 2021-11-20T07:35:04.312 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.305+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd8105f30 con 0x7fecd80f51a0 2021-11-20T07:35:04.313 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.306+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecd401e420 con 0x7fecd80f51a0 2021-11-20T07:35:04.313 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.306+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8045ad0 con 0x7fecd80f51a0 2021-11-20T07:35:04.313 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.306+0000 7fecce7fc700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 0x7fecc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:04.314 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.306+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecd4040ff0 con 0x7fecd80f51a0 2021-11-20T07:35:04.314 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.306+0000 7fecdd1ea700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 0x7fecc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:04.314 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.307+0000 7fecdd1ea700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 0x7fecc40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fecc80096a0 tx=0x7fecc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:04.314 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.311+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecd406d2b0 con 0x7fecd80f51a0 2021-11-20T07:35:04.532 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.530+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecd8106d80 con 0x7fecd80f51a0 2021-11-20T07:35:04.533 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.531+0000 7fecce7fc700 1 -- 172.21.15.112:0/1629298988 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecd40095f0 con 0x7fecd80f51a0 2021-11-20T07:35:04.533 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:35:04.534 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","modified":"2021-11-20T07:33:19.655860Z","created":"2021-11-20T07:33:19.655860Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi071","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T07:35:04.535 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 msgr2=0x7fecc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:04.536 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 0x7fecc40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fecc80096a0 tx=0x7fecc8006b40).stop 2021-11-20T07:35:04.536 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 msgr2=0x7fecd81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:04.536 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fecd4009830 tx=0x7fecd400b040).stop 2021-11-20T07:35:04.537 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 shutdown_connections 2021-11-20T07:35:04.537 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fecc402ff10 0x7fecc40323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:04.537 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 --2- 172.21.15.112:0/1629298988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecd80f51a0 0x7fecd81053b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:04.537 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 >> 172.21.15.112:0/1629298988 conn(0x7fecd80f09a0 msgr2=0x7fecd80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:04.537 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 shutdown_connections 2021-11-20T07:35:04.538 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:04.533+0000 7fecde9ed700 1 -- 172.21.15.112:0/1629298988 wait complete. 2021-11-20T07:35:04.538 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2021-11-20T07:35:04.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11-20T07:35:03.540680+0000 mon.smithi071 (mon.0 2021-11-20T07:35:04.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: ) 145 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:04.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11-20T07:35:03.541967+0000 mon.smithi071 (mon.0) 146 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.qaikrr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T07:35:04.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:04.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: -20T07:35:03.546800+0000 mon.smithi071 (mon.0) 147 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.qaikrr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-11-20T07:35:04.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11-20T07:35:03.547661+0000 2021-11-20T07:35:04.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: mon.smithi071 (mon.0) 148 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T07:35:04.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11-20T07:35:04.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: 2021-11-20T07:35:03.548592+0000 mon.smithi071 (mon.0) 149 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:04.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: cephadm 2021-11-20T07:35 2021-11-20T07:35:04.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: :03.549437+0000 mgr.smithi071.ynqmwn (mgr.14162) 31 : cephadm [INF] Deploying daemon mgr.smithi112.qaikrr on smithi112 2021-11-20T07:35:04.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: cluster 2021-11-20T07:35:04 2021-11-20T07:35:04.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: .328915+0000 mgr.smithi071.ynqmwn (mgr.14162) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:04.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: audit 2021-11-20T07:35: 2021-11-20T07:35:04.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:04 smithi071 conmon[33687]: 04.532838+0000 mon.smithi071 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.112:0/1629298988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:35:06.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T07:35:06.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T07:35:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:07 smithi071 conmon[33687]: cluster 2021-11-20T07:35:06.329293+0000 mgr.smithi071.ynqmwn (mgr.14162) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:07 smithi071 conmon[33687]: audit 2021-11-20T07:35:06.359282+0000 mon.smithi071 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:07 smithi071 conmon[33687]: audit 2021-11-20T07:35:06.360416+0000 mon.smithi071 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T07:35:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:07 smithi071 conmon[33687]: audit 2021-11-20T07:35:06.361243+0000 mon.smithi071 (mon.0) 153 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:07.936 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 -- 172.21.15.112:0/2158996151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 msgr2=0x7fd1e00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:07.936 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/2158996151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e00f5750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd1d8009960 tx=0x7fd1d8004d10).stop 2021-11-20T07:35:07.936 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 -- 172.21.15.112:0/2158996151 shutdown_connections 2021-11-20T07:35:07.936 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/2158996151 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e00f5750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 -- 172.21.15.112:0/2158996151 >> 172.21.15.112:0/2158996151 conn(0x7fd1e00f0a90 msgr2=0x7fd1e00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 -- 172.21.15.112:0/2158996151 shutdown_connections 2021-11-20T07:35:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.933+0000 7fd1e4db7700 1 -- 172.21.15.112:0/2158996151 wait complete. 2021-11-20T07:35:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1e4db7700 1 Processor -- start 2021-11-20T07:35:07.938 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1e4db7700 1 -- start start 2021-11-20T07:35:07.938 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1e4db7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:07.938 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1e4db7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01074e0 con 0x7fd1e00f5330 2021-11-20T07:35:07.938 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1de59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:07.938 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1de59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.112:43622/0 (socket says 172.21.15.112:43622) 2021-11-20T07:35:07.939 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1de59c700 1 -- 172.21.15.112:0/4209307996 learned_addr learned my addr 172.21.15.112:0/4209307996 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:35:07.939 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.934+0000 7fd1de59c700 1 -- 172.21.15.112:0/4209307996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8009640 con 0x7fd1e00f5330 2021-11-20T07:35:07.939 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1de59c700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd1d8004b70 tx=0x7fd1d8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:07.939 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d8018070 con 0x7fd1e00f5330 2021-11-20T07:35:07.940 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e0060220 con 0x7fd1e00f5330 2021-11-20T07:35:07.940 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1d800b870 con 0x7fd1e00f5330 2021-11-20T07:35:07.940 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1d801c560 con 0x7fd1e00f5330 2021-11-20T07:35:07.940 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.935+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e010bfe0 con 0x7fd1e00f5330 2021-11-20T07:35:07.941 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1d8013630 con 0x7fd1e00f5330 2021-11-20T07:35:07.942 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e0045ad0 con 0x7fd1e00f5330 2021-11-20T07:35:07.942 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1d6ffd700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:07.942 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1d8041290 con 0x7fd1e00f5330 2021-11-20T07:35:07.942 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1ddd9b700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:07.943 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.936+0000 7fd1ddd9b700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 0x7fd1cc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:07.943 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:07.940+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1d8004810 con 0x7fd1e00f5330 2021-11-20T07:35:08.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.157+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1e005ea50 con 0x7fd1e00f5330 2021-11-20T07:35:08.159 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.158+0000 7fd1d6ffd700 1 -- 172.21.15.112:0/4209307996 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1d8012380 con 0x7fd1e00f5330 2021-11-20T07:35:08.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:35:08.160 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","modified":"2021-11-20T07:33:19.655860Z","created":"2021-11-20T07:33:19.655860Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi071","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T07:35:08.161 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.159+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 msgr2=0x7fd1cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:08.161 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.159+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 0x7fd1cc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).stop 2021-11-20T07:35:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.159+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 msgr2=0x7fd1e010baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.159+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd1d8004b70 tx=0x7fd1d8004d10).stop 2021-11-20T07:35:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 shutdown_connections 2021-11-20T07:35:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd1cc02ff60 0x7fd1cc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 --2- 172.21.15.112:0/4209307996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1e00f5330 0x7fd1e010baf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:08.163 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 >> 172.21.15.112:0/4209307996 conn(0x7fd1e00f0a90 msgr2=0x7fd1e00f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:08.163 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 shutdown_connections 2021-11-20T07:35:08.163 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:08.160+0000 7fd1e4db7700 1 -- 172.21.15.112:0/4209307996 wait complete. 2021-11-20T07:35:08.163 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2021-11-20T07:35:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:08 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:06.362176+0000 mgr.smithi071.ynqmwn (mgr.14162) 34 : cephadm [INF] Deploying daemon mon.smithi112 on smithi112 2021-11-20T07:35:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:08 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:35:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:08 smithi071 conmon[33687]: 07:35:08.158479+0000 mon.smithi071 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.112:0/4209307996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:35:09.592 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T07:35:09.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T07:35:09.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:09 smithi071 conmon[33687]: cluster 2021-11-20T07:35:08.329688+0000 mgr.smithi071.ynqmwn (mgr.14162) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:10.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:09 smithi112 systemd[1]: Starting Ceph mon.smithi112 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:35:11.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:11 smithi071 conmon[33687]: cluster 2021-11-20T07:35:10.330105+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:35:11.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:11 smithi071 conmon[33687]: 36 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:12.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:12 smithi112 podman[35920]: 2021-11-20T07:35:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: cluster 2021-11-20T07:35:11.408850+0000 mon.smithi071 (mon.0) 155 : cluster [DBG] Standby manager daemon smithi112.qaikrr started 2021-11-20T07:35:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:35:12.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: 35:11.410518+0000 mon.smithi071 (mon.0) 156 : audit [DBG] from='mgr.? 172.21.15.112:0/2329959413' entity='mgr.smithi112.qaikrr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi112.qaikrr/crt"}]: dispatch 2021-11-20T07:35:12.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: audit 2021-11-20T07:35:11.411202+0000 mon.smithi071 (mon.0) 157 : audit [DBG] from='mgr.? 172.21.15.112:0/2329959413' entity='mgr.smithi112.qaikrr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T07:35:12.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: audit 2021-11-20T07:35:11.413163+0000 mon.smithi071 (mon.0) 158 : audit [DBG] from='mgr.? 172.21.15.112:0/2329959413' entity='mgr.smithi112.qaikrr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi112.qaikrr/key"}]: dispatch 2021-11-20T07:35:12.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: audit 2021-11-20T07:35:11.414086+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:12.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:12 smithi071 conmon[33687]: 159 : audit [DBG] from='mgr.? 172.21.15.112:0/2329959413' entity='mgr.smithi112.qaikrr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T07:35:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:13 smithi071 conmon[33687]: cluster 2021-11-20T07:35:12.328887+0000 mon.smithi071 (mon.0) 160 : cluster [DBG] mgrmap e17: smithi071.ynqmwn(active, since 56s), standbys: smithi112.qaikrr 2021-11-20T07:35:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:13 smithi071 conmon[33687]: audit 2021-11-20T07:35:12. 2021-11-20T07:35:13.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:13 smithi071 conmon[33687]: 329080+0000 mon.smithi071 (mon.0) 161 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi112.qaikrr", "id": "smithi112.qaikrr"}]: dispatch 2021-11-20T07:35:13.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:13 smithi071 conmon[33687]: cluster 2021-11-20T07:35:12.330411+0000 mgr.smithi071.ynqmwn (mgr.14162) 37 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:14.010 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.843+0000 7f148ad2f700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T07:35:14.010 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.843+0000 7f148ad2f700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T07:35:14.010 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.843+0000 7f148ad2f700 0 pidfile_write: ignore empty --pid-file 2021-11-20T07:35:14.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.847+0000 7f148ad2f700 0 load: jerasure load: lrc load: isa 2021-11-20T07:35:14.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T07:35:14.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:14.012 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T07:35:14.012 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Compile date May 13 2021 2021-11-20T07:35:14.012 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: DB SUMMARY 2021-11-20T07:35:14.012 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T07:35:14.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T07:35:14.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-11-20T07:35:14.014 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.014 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi112/store.db dir, Total Num: 0, files: 2021-11-20T07:35:14.014 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.014 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi112/store.db: 000003.log size: 511 ; 2021-11-20T07:35:14.015 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.015 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T07:35:14.015 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T07:35:14.015 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T07:35:14.015 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.env: 0x5626a03621c0 2021-11-20T07:35:14.016 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 bash[35798]: b31f773ae07f8a47f8ea709ded3e1e811268aefce76fbcb8e3e9b0f82bea52c4 2021-11-20T07:35:14.016 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.016 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.fs: Posix File System 2021-11-20T07:35:14.016 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.info_log: 0x5626a1a45960 2021-11-20T07:35:14.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T07:35:14.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.statistics: (nil) 2021-11-20T07:35:14.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.use_fsync: 0 2021-11-20T07:35:14.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T07:35:14.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:14.018 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T07:35:14.018 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T07:35:14.018 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T07:35:14.018 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T07:35:14.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T07:35:14.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T07:35:14.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T07:35:14.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T07:35:14.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T07:35:14.020 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T07:35:14.020 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.db_log_dir: 2021-11-20T07:35:14.020 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi112/store.db 2021-11-20T07:35:14.020 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T07:35:14.020 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T07:35:14.021 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T07:35:14.021 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T07:35:14.021 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T07:35:14.021 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T07:35:14.022 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T07:35:14.022 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T07:35:14.022 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T07:35:14.022 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T07:35:14.022 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.write_buffer_manager: 0x5626a26ba030 2021-11-20T07:35:14.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T07:35:14.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T07:35:14.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T07:35:14.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T07:35:14.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T07:35:14.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T07:35:14.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T07:35:14.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T07:35:14.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T07:35:14.025 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.unordered_write: 0 2021-11-20T07:35:14.025 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.025 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T07:35:14.025 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T07:35:14.025 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T07:35:14.026 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T07:35:14.026 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.row_cache: None 2021-11-20T07:35:14.026 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.wal_filter: None 2021-11-20T07:35:14.026 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T07:35:14.027 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T07:35:14.027 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T07:35:14.027 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T07:35:14.027 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T07:35:14.027 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T07:35:14.028 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:14.028 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T07:35:14.028 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T07:35:14.028 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T07:35:14.028 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T07:35:14.029 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T07:35:14.029 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T07:35:14.029 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T07:35:14.029 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T07:35:14.029 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T07:35:14.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T07:35:14.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T07:35:14.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T07:35:14.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T07:35:14.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T07:35:14.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T07:35:14.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_open_files: -1 2021-11-20T07:35:14.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T07:35:14.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T07:35:14.032 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T07:35:14.032 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T07:35:14.032 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Compression algorithms supported: 2021-11-20T07:35:14.032 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T07:35:14.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kZSTD supported: 0 2021-11-20T07:35:14.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T07:35:14.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T07:35:14.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T07:35:14.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T07:35:14.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T07:35:14.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T07:35:14.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T07:35:14.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi112/store.db/MANIFEST-000001 2021-11-20T07:35:14.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:14.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T07:35:14.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T07:35:14.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.merge_operator: 2021-11-20T07:35:14.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_filter: None 2021-11-20T07:35:14.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T07:35:14.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T07:35:14.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T07:35:14.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5626a195c128) 2021-11-20T07:35:14.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: cache_index_and_filter_blocks: 1 2021-11-20T07:35:14.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T07:35:14.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T07:35:14.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: pin_top_level_index_and_filter: 1 2021-11-20T07:35:14.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: index_type: 0 2021-11-20T07:35:14.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: data_block_index_type: 0 2021-11-20T07:35:14.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: index_shortening: 1 2021-11-20T07:35:14.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T07:35:14.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: hash_index_allow_collision: 1 2021-11-20T07:35:14.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: checksum: 1 2021-11-20T07:35:14.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: no_block_cache: 0 2021-11-20T07:35:14.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_cache: 0x5626a1992f10 2021-11-20T07:35:14.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_cache_name: BinnedLRUCache 2021-11-20T07:35:14.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_cache_options: 2021-11-20T07:35:14.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: capacity : 536870912 2021-11-20T07:35:14.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: num_shard_bits : 4 2021-11-20T07:35:14.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: strict_capacity_limit : 0 2021-11-20T07:35:14.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: high_pri_pool_ratio: 0.000 2021-11-20T07:35:14.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_cache_compressed: (nil) 2021-11-20T07:35:14.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: persistent_cache: (nil) 2021-11-20T07:35:14.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_size: 4096 2021-11-20T07:35:14.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_size_deviation: 10 2021-11-20T07:35:14.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_restart_interval: 16 2021-11-20T07:35:14.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: index_block_restart_interval: 1 2021-11-20T07:35:14.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: metadata_block_size: 4096 2021-11-20T07:35:14.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: partition_filters: 0 2021-11-20T07:35:14.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: use_delta_encoding: 1 2021-11-20T07:35:14.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T07:35:14.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: whole_key_filtering: 1 2021-11-20T07:35:14.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: verify_compression: 0 2021-11-20T07:35:14.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: read_amp_bytes_per_bit: 0 2021-11-20T07:35:14.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: format_version: 2 2021-11-20T07:35:14.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: enable_index_compression: 1 2021-11-20T07:35:14.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: block_align: 0 2021-11-20T07:35:14.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T07:35:14.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T07:35:14.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression: NoCompression 2021-11-20T07:35:14.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T07:35:14.047 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T07:35:14.047 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T07:35:14.047 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.num_levels: 7 2021-11-20T07:35:14.047 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T07:35:14.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T07:35:14.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T07:35:14.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T07:35:14.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T07:35:14.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T07:35:14.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T07:35:14.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:35:14.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T07:35:14.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T07:35:14.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T07:35:14.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T07:35:14.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T07:35:14.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T07:35:14.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T07:35:14.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T07:35:14.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T07:35:14.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T07:35:14.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T07:35:14.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T07:35:14.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T07:35:14.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T07:35:14.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T07:35:14.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T07:35:14.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T07:35:14.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T07:35:14.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T07:35:14.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T07:35:14.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T07:35:14.054 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.054 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T07:35:14.054 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T07:35:14.054 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T07:35:14.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T07:35:14.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T07:35:14.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T07:35:14.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T07:35:14.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T07:35:14.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T07:35:14.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T07:35:14.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T07:35:14.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T07:35:14.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T07:35:14.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T07:35:14.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T07:35:14.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T07:35:14.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T07:35:14.058 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T07:35:14.058 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T07:35:14.058 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T07:35:14.058 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T07:35:14.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T07:35:14.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T07:35:14.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T07:35:14.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T07:35:14.060 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T07:35:14.060 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T07:35:14.060 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T07:35:14.060 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T07:35:14.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T07:35:14.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.ttl: 2592000 2021-11-20T07:35:14.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.848+0000 7f148ad2f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T07:35:14.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.849+0000 7f148ad2f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi112/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T07:35:14.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:14.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:13.849+0000 7f148ad2f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T07:35:14.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.849+0000 7f148ad2f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393713850139, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-11-20T07:35:14.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.849+0000 7f148ad2f700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-11-20T07:35:14.063 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.849+0000 7f148ad2f700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T07:35:14.063 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.850+0000 7f148ad2f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393713850668, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637393713, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T07:35:14.063 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.850+0000 7f148ad2f700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-11-20T07:35:14.063 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.064 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.850+0000 7f148ad2f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637393713851071, "job": 1, "event": "recovery_finished"} 2021-11-20T07:35:14.064 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.851+0000 7f148ad2f700 4 rocksdb: DB pointer 0x5626a1a57800 2021-11-20T07:35:14.064 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.851+0000 7f1471b47700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T07:35:14.064 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.851+0000 7f1471b47700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T07:35:14.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** DB Stats ** 2021-11-20T07:35:14.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:35:14.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T07:35:14.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T07:35:14.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:35:14.066 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T07:35:14.066 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T07:35:14.066 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T07:35:14.066 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.066 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** Compaction Stats [default] ** 2021-11-20T07:35:14.067 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:35:14.067 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:35:14.067 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.067 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.068 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.068 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.068 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** Compaction Stats [default] ** 2021-11-20T07:35:14.068 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:35:14.068 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:35:14.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:35:14.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:35:14.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:35:14.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:35:14.070 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:35:14.070 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:35:14.070 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:35:14.070 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:35:14.071 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:35:14.071 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.071 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:35:14.071 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.071 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** Compaction Stats [default] ** 2021-11-20T07:35:14.072 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:35:14.072 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:35:14.072 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.072 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T07:35:14.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** Compaction Stats [default] ** 2021-11-20T07:35:14.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T07:35:14.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T07:35:14.074 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2021-11-20T07:35:14.074 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T07:35:14.074 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T07:35:14.074 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T07:35:14.074 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T07:35:14.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T07:35:14.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T07:35:14.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:35:14.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T07:35:14.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T07:35:14.076 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.076 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T07:35:14.076 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.076 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.851+0000 7f148ad2f700 0 mon.smithi112 does not exist in monmap, will attempt to join an existing cluster 2021-11-20T07:35:14.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.851+0000 7f148ad2f700 0 using public_addr v2:172.21.15.112:0/0 -> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2021-11-20T07:35:14.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.852+0000 7f148ad2f700 0 starting mon.smithi112 rank -1 at public addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] at bind addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi112 fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:35:14.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.853+0000 7f148ad2f700 1 mon.smithi112@-1(???) e0 preinit fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:35:14.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).mds e1 new map 2021-11-20T07:35:14.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).mds e1 print_map 2021-11-20T07:35:14.078 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: e1 2021-11-20T07:35:14.078 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T07:35:14.078 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:35:14.078 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: legacy client fscid: -1 2021-11-20T07:35:14.079 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.079 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: No filesystems configured 2021-11-20T07:35:14.079 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 2021-11-20T07:35:14.079 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-11-20T07:35:14.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-11-20T07:35:14.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-11-20T07:35:14.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-11-20T07:35:14.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-11-20T07:35:14.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-11-20T07:35:14.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.881+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2021-11-20T07:35:14.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 systemd[1]: Started Ceph mon.smithi112 for 16910ea2-49d4-11ec-8c2c-001a4aab830c. 2021-11-20T07:35:14.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.882+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T07:35:14.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.882+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:35:14.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.882+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:35:14.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.882+0000 7f147434c700 0 mon.smithi112@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T07:35:14.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:35:14.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: 07:35:12.328887+0000 mon.smithi071 (mon.0) 160 : cluster [DBG] mgrmap e17: smithi071.ynqmwn(active, since 56s), standbys: smithi112.qaikrr 2021-11-20T07:35:14.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: audit 2021 2021-11-20T07:35:14.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: -11-20T07:35:12.329080+0000 mon.smithi071 (mon.0) 161 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr metadata", "who": "smithi112.qaikrr", "id": "smithi112.qaikrr"}]: dispatch 2021-11-20T07:35:14.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: cluster 2021-11-20T07:35:12.330411+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:14.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: mgr.14162) 37 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:14.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.882+0000 7f147434c700 1 mon.smithi112@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2021-11-20T07:35:14.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expand map: {default=false} 2021-11-20T07:35:14.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta from 'false' to 'false' 2021-11-20T07:35:14.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expanded map: {default=false} 2021-11-20T07:35:14.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expand map: {default=info} 2021-11-20T07:35:14.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta from 'info' to 'info' 2021-11-20T07:35:14.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expanded map: {default=info} 2021-11-20T07:35:14.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expand map: {default=daemon} 2021-11-20T07:35:14.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-20T07:35:14.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-20T07:35:14.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expand map: {default=debug} 2021-11-20T07:35:14.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-20T07:35:14.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 20 expand_channel_meta expanded map: {default=debug} 2021-11-20T07:35:14.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:13 smithi112 conmon[36062]: debug 2021-11-20T07:35:13.884+0000 7f147434c700 10 mon.smithi112@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2021-11-20T07:35:19.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:13.890923+0000 mon.smithi071 (mon.0) 163 : audit [DBG] 2021-11-20T07:35:19.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:35:19.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:13.891101+0000 mon.smithi071 (mon.0) 164 : cluster [INF] mon.smithi071 calling monitor election 2021-11-20T07:35:19.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:13.905046+0000 mon.smithi071 (mon.0) 165 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:14.330671+0000 mgr.smithi071.ynqmwn (mgr.14162) 38 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:14.887035+0000 mon.smithi071 (mon.0) 166 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:15.886999+0000 mon.smithi071 (mon.0) 167 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:15.888594+0000 mon.smithi112 (mon.1) 1 : cluster [INF] mon.smithi112 calling monitor election 2021-11-20T07:35:19.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:16.330997+0000 mgr.smithi071.ynqmwn (mgr.14162) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:16.454166+0000 mon.smithi071 (mon.0) 168 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:35:19.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:16.459721+0000 mon.smithi071 (mon.0) 169 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:35:19.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:16.887176+0000 mon.smithi071 (mon.0) 170 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:17.887492+0000 mon.smithi071 (mon.0) 171 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.331409+0000 mgr.smithi071.ynqmwn (mgr.14162) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:18.887446+0000 mon.smithi071 (mon.0) 172 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.918727+0000 mon.smithi071 (mon.0) 173 : cluster [INF] mon.smithi071 is new leader, mons smithi071,smithi112 in quorum (ranks 0,1) 2021-11-20T07:35:19.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.922054+0000 mon.smithi071 (mon.0) 174 : cluster [DBG] monmap e2: 2 mons at {smithi071=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],smithi112=[v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0]} 2021-11-20T07:35:19.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.922098+0000 mon.smithi071 (mon.0) 175 : cluster [DBG] fsmap 2021-11-20T07:35:19.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.922126+0000 mon.smithi071 (mon.0) 176 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T07:35:19.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.922523+0000 mon.smithi071 (mon.0) 177 : cluster [DBG] mgrmap e17: smithi071.ynqmwn(active, since 62s), standbys: smithi112.qaikrr 2021-11-20T07:35:19.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: cluster 2021-11-20T07:35:18.926484+0000 mon.smithi071 (mon.0) 178 : cluster [INF] overall HEALTH_OK 2021-11-20T07:35:19.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:18 smithi112 conmon[36062]: audit 2021-11-20T07:35:18.929182+0000 mon.smithi071 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:19.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35:13.890923+0000 mon.smithi071 (mon.0) 163 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi071"}]: dispatch 2021-11-20T07:35:19.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11-20T07:35:13.891101+0000 mon.smithi071 (mon.0) 164 : cluster [INF] mon.smithi071 calling monitor election 2021-11-20T07:35:19.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35:13.905046+0000 mon.smithi071 (mon.0) 165 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11-20T07:35:14.330671 2021-11-20T07:35:19.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 38 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:19.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35:19.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:14.887035+0000 mon.smithi071 ( 2021-11-20T07:35:19.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.0) 166 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:35:19.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 20T07:35:15.886999+0000 mon.smithi071 ( 2021-11-20T07:35:19.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.0) 167 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:35:19.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 11-20T07:35:15.888594+0000 2021-11-20T07:35:19.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.smithi112 (mon.1) 1 : cluster [INF] mon.smithi112 calling monitor election 2021-11-20T07:35:19.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:35:19.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: -20T07:35:16.330997+0000 mgr.smithi071.ynqmwn 2021-11-20T07:35:19.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: (mgr.14162) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:35:19.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 11-20T07:35:16.454166+0000 2021-11-20T07:35:19.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.smithi071 (mon.0) 168 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:35:19.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:19.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35:19.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:16.459721+0000 mon.smithi071 2021-11-20T07:35:19.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: (mon.0) 169 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:35:19.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:19.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: -20T07:35:16.887176+0000 mon.smithi071 (mon.0) 170 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35:17.887492+0000 mon.smithi071 (mon.0) 171 : audit [DBG] 2021-11-20T07:35:19.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:35:19.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: -11-20T07:35:18.331409+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:35:19.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:19.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:35:19.326 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 07:35:18.887446+0000 mon.smithi071 (mon.0 2021-11-20T07:35:19.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: ) 172 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:19.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11-20T07 2021-11-20T07:35:19.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: :35:18.918727+0000 mon.smithi071 (mon.0 2021-11-20T07:35:19.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: ) 173 : cluster [INF] mon.smithi071 is new leader, mons smithi071,smithi112 in quorum (ranks 0,1) 2021-11-20T07:35:19.327 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:35:19.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 20T07:35:18.922054+0000 mon.smithi071 ( 2021-11-20T07:35:19.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.0) 174 : cluster [DBG] monmap e2: 2 mons at {smithi071=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],smithi112=[v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0]} 2021-11-20T07:35:19.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:35:19.328 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: -11-20T07:35:18.922098+0000 2021-11-20T07:35:19.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.smithi071 (mon.0) 175 : cluster [DBG] fsmap 2021-11-20T07:35:19.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:19.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11-20T07:35:19.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: 2021-11-20T07:35:18.922126+0000 mon.smithi071 ( 2021-11-20T07:35:19.329 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.0) 176 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T07:35:19.330 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:35:19.330 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: -11-20T07:35:18.922523+0000 2021-11-20T07:35:19.330 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mon.smithi071 (mon.0) 177 : cluster [DBG] 2021-11-20T07:35:19.330 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: mgrmap e17: smithi071.ynqmwn(active, since 62s), standbys: smithi112.qaikrr 2021-11-20T07:35:19.331 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: cluster 2021-11-20T07:35:18 2021-11-20T07:35:19.331 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: .926484+0000 mon.smithi071 (mon.0) 178 : cluster 2021-11-20T07:35:19.331 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: [INF] overall HEALTH_OK 2021-11-20T07:35:19.331 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: audit 2021-11-20T07:35 2021-11-20T07:35:19.331 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: :18.929182+0000 mon.smithi071 (mon.0) 179 2021-11-20T07:35:19.332 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:18 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:20.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:19 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:18.930251+0000 mgr.smithi071.ynqmwn (mgr.14162) 41 : cephadm [INF] Deploying daemon node-exporter.smithi112 on smithi112 2021-11-20T07:35:20.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:19 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:35:20.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:19 smithi112 conmon[36062]: 20T07:35:19.887466+0000 mon.smithi071 (mon.0) 180 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:20.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:19 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:18.930251+0000 mgr.smithi071.ynqmwn (mgr.14162) 41 : cephadm 2021-11-20T07:35:20.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:19 smithi071 conmon[33687]: [INF] Deploying daemon node-exporter.smithi112 on smithi112 2021-11-20T07:35:20.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:19 smithi071 conmon[33687]: audit 2021-11-20T07:35:19.887466+0000 mon.smithi071 (mon.0) 180 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2021-11-20T07:35:20.329 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 -- 172.21.15.112:0/4178662646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7940003250 msgr2=0x7f79400056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:20.329 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 --2- 172.21.15.112:0/4178662646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7940003250 0x7f79400056e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f795805e3c0 tx=0x7f7948002040).stop 2021-11-20T07:35:20.329 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 -- 172.21.15.112:0/4178662646 shutdown_connections 2021-11-20T07:35:20.330 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 --2- 172.21.15.112:0/4178662646 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7940003250 0x7f79400056e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:20.330 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 -- 172.21.15.112:0/4178662646 >> 172.21.15.112:0/4178662646 conn(0x7f79580efc60 msgr2=0x7f79580f20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:20.330 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 -- 172.21.15.112:0/4178662646 shutdown_connections 2021-11-20T07:35:20.331 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.323+0000 7f795da59700 1 -- 172.21.15.112:0/4178662646 wait complete. 2021-11-20T07:35:20.331 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 Processor -- start 2021-11-20T07:35:20.331 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 -- start start 2021-11-20T07:35:20.331 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:20.332 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 0x7f79580f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:20.332 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580f6400 con 0x7f79580f5a00 2021-11-20T07:35:20.332 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f795da59700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580f6540 con 0x7f7940003250 2021-11-20T07:35:20.332 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f7956ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:20.333 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f7956ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:58080/0 (socket says 172.21.15.112:58080) 2021-11-20T07:35:20.333 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f7956ffd700 1 -- 172.21.15.112:0/3524235936 learned_addr learned my addr 172.21.15.112:0/3524235936 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:35:20.333 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f7956ffd700 1 -- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 msgr2=0x7f79580f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-11-20T07:35:20.333 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f79567fc700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 0x7f79580f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:20.334 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.325+0000 7f7956ffd700 1 -- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 msgr2=0x7f79580f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-20T07:35:20.334 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f7956ffd700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:35:20.334 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f7956ffd700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:35:20.334 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f79567fc700 1 -- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 msgr2=0x7f79580f7580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T07:35:20.335 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f79567fc700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:20.335 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f79567fc700 1 -- 172.21.15.112:0/3524235936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948005040 con 0x7f79580f5a00 2021-11-20T07:35:20.335 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f79567fc700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 0x7f79580f5e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f794c009390 tx=0x7f794c006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:20.335 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f794c00e650 con 0x7f79580f5a00 2021-11-20T07:35:20.335 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f794c009e40 con 0x7f79580f5a00 2021-11-20T07:35:20.336 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79580f67f0 con 0x7f79580f5a00 2021-11-20T07:35:20.336 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f794c011b80 con 0x7f79580f5a00 2021-11-20T07:35:20.336 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.326+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795810e900 con 0x7f79580f5a00 2021-11-20T07:35:20.336 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.327+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958059100 con 0x7f79580f5a00 2021-11-20T07:35:20.337 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.327+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f794c018030 con 0x7f79580f5a00 2021-11-20T07:35:20.337 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.328+0000 7f793f7fe700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 0x7f794405e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:20.337 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.328+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f794c074920 con 0x7f79580f5a00 2021-11-20T07:35:20.337 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.328+0000 7f7956ffd700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 0x7f794405e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:20.338 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.328+0000 7f7956ffd700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 0x7f794405e8c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7948007850 tx=0x7f7948002040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:20.338 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.332+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f794c0452b0 con 0x7f79580f5a00 2021-11-20T07:35:20.571 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.570+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7958045ad0 con 0x7f79580f5a00 2021-11-20T07:35:20.572 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.571+0000 7f793f7fe700 1 -- 172.21.15.112:0/3524235936 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7f794c016030 con 0x7f79580f5a00 2021-11-20T07:35:20.572 INFO:teuthology.orchestra.run.smithi112.stdout: 2021-11-20T07:35:20.572 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":2,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","modified":"2021-11-20T07:35:13.888026Z","created":"2021-11-20T07:33:19.655860Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi071","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-11-20T07:35:20.574 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 msgr2=0x7f794405e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:20.574 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 0x7f794405e8c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7948007850 tx=0x7f7948002040).stop 2021-11-20T07:35:20.574 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 msgr2=0x7f79580f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:20.574 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 0x7f79580f5e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f794c009390 tx=0x7f794c006e20).stop 2021-11-20T07:35:20.575 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 shutdown_connections 2021-11-20T07:35:20.575 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7940003250 0x7f79580f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:20.576 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f794405c400 0x7f794405e8c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:20.576 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 --2- 172.21.15.112:0/3524235936 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79580f5a00 0x7f79580f5e80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:20.576 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 >> 172.21.15.112:0/3524235936 conn(0x7f79580efc60 msgr2=0x7f79580facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:20.576 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 shutdown_connections 2021-11-20T07:35:20.577 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:35:20.573+0000 7f795da59700 1 -- 172.21.15.112:0/3524235936 wait complete. 2021-11-20T07:35:20.577 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 2 2021-11-20T07:35:21.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:21 smithi112 conmon[36062]: cluster 2021-11-20T07:35:20.331729+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:35:21.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:21 smithi112 conmon[36062]: 14162) 42 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:21.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:21 smithi112 conmon[36062]: audit 2021-11-20T07:35:20.571582+0000 mon.smithi071 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.112:0/3524235936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:35:21.613 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-11-20T07:35:21.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph config generate-minimal-conf 2021-11-20T07:35:21.646 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:21 smithi071 conmon[33687]: cluster 2021-11-20T07:35:20.331729+0000 mgr.smithi071.ynqmwn (mgr.14162) 42 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:21.646 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:21 smithi071 conmon[33687]: 2021-11-20T07:35:21.646 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:21 smithi071 conmon[33687]: audit 2021-11-20T07:35:20.571582+0000 mon.smithi071 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.112:0/3524235936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T07:35:23.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:23 smithi071 conmon[33687]: audit 2021-11-20T07:35:22.269988+0000 mon.smithi071 (mon.0) 182 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:23.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:23 smithi071 conmon[33687]: audit 2021-11-20T07:35:22.270822+0000 mon.smithi071 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:35:23.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:23 smithi071 conmon[33687]: cluster 2021-11-20T07:35:22.331980+0000 mgr.smithi071.ynqmwn (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:23.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:23 smithi112 conmon[36062]: audit 2021-11-20T07:35:22.269988+0000 mon.smithi071 (mon. 2021-11-20T07:35:23.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:23 smithi112 conmon[36062]: 0) 182 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:23.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:23 smithi112 conmon[36062]: audit 2021-11-20T07:35:22.270822+0000 mon.smithi071 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:35:23.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:23 smithi112 conmon[36062]: cluster 2021-11-20T07:35:22.331980+0000 mgr.smithi071.ynqmwn (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:23.977 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 -- 172.21.15.71:0/2225886249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc440f5410 msgr2=0x7fbc440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:23.977 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 --2- 172.21.15.71:0/2225886249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc440f5410 0x7fbc440f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fbc34009fb0 tx=0x7fbc34009b30).stop 2021-11-20T07:35:23.978 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 -- 172.21.15.71:0/2225886249 shutdown_connections 2021-11-20T07:35:23.978 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 --2- 172.21.15.71:0/2225886249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc440f5410 0x7fbc440f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:23.978 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 -- 172.21.15.71:0/2225886249 >> 172.21.15.71:0/2225886249 conn(0x7fbc440f0b70 msgr2=0x7fbc440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:23.978 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 -- 172.21.15.71:0/2225886249 shutdown_connections 2021-11-20T07:35:23.979 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.974+0000 7fbc4896e700 1 -- 172.21.15.71:0/2225886249 wait complete. 2021-11-20T07:35:23.979 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 Processor -- start 2021-11-20T07:35:23.979 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 -- start start 2021-11-20T07:35:23.979 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 0x7fbc441056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:23.980 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:23.980 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441060f0 con 0x7fbc44105b90 2021-11-20T07:35:23.980 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4896e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44106230 con 0x7fbc440f5410 2021-11-20T07:35:23.980 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:23.980 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47282/0 (socket says 172.21.15.71:47282) 2021-11-20T07:35:23.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 -- 172.21.15.71:0/3938331074 learned_addr learned my addr 172.21.15.71:0/3938331074 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:35:23.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4259c700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 0x7fbc441056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:23.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 -- 172.21.15.71:0/3938331074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 msgr2=0x7fbc441056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:23.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 0x7fbc441056a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:23.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 -- 172.21.15.71:0/3938331074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc34005040 con 0x7fbc44105b90 2021-11-20T07:35:23.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc4259c700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 0x7fbc441056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:35:23.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.975+0000 7fbc41d9b700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbc380093c0 tx=0x7fbc38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:23.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.976+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc3800e650 con 0x7fbc44105b90 2021-11-20T07:35:23.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.976+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc44107090 con 0x7fbc44105b90 2021-11-20T07:35:23.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.976+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc38009e70 con 0x7fbc44105b90 2021-11-20T07:35:23.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.976+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc38011c30 con 0x7fbc44105b90 2021-11-20T07:35:23.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.976+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc44107650 con 0x7fbc44105b90 2021-11-20T07:35:23.984 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.977+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44059100 con 0x7fbc44105b90 2021-11-20T07:35:23.984 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.977+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbc38018030 con 0x7fbc44105b90 2021-11-20T07:35:23.985 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.978+0000 7fbc32ffd700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 0x7fbc2c05e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:23.985 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.978+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc38074a10 con 0x7fbc44105b90 2021-11-20T07:35:23.985 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.978+0000 7fbc4259c700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 0x7fbc2c05e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:23.985 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.978+0000 7fbc4259c700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 0x7fbc2c05e870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbc34004dd0 tx=0x7fbc3400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:23.986 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:23.982+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc380452b0 con 0x7fbc44105b90 2021-11-20T07:35:24.138 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.136+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbc44045ad0 con 0x7fbc44105b90 2021-11-20T07:35:24.138 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.136+0000 7fbc32ffd700 1 -- 172.21.15.71:0/3938331074 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7fbc38016020 con 0x7fbc44105b90 2021-11-20T07:35:24.138 INFO:teuthology.orchestra.run.smithi071.stdout:# minimal ceph.conf for 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:35:24.139 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2021-11-20T07:35:24.139 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 16910ea2-49d4-11ec-8c2c-001a4aab830c 2021-11-20T07:35:24.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon_host = [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] 2021-11-20T07:35:24.140 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 msgr2=0x7fbc2c05e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:24.140 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 0x7fbc2c05e870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbc34004dd0 tx=0x7fbc3400b040).stop 2021-11-20T07:35:24.141 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 msgr2=0x7fbc44106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:24.141 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbc380093c0 tx=0x7fbc38006ee0).stop 2021-11-20T07:35:24.141 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 shutdown_connections 2021-11-20T07:35:24.141 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc440f5410 0x7fbc441056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:24.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc2c05c3b0 0x7fbc2c05e870 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:24.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 --2- 172.21.15.71:0/3938331074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc44105b90 0x7fbc44106a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:24.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.138+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 >> 172.21.15.71:0/3938331074 conn(0x7fbc440f0b70 msgr2=0x7fbc440f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:24.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.139+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 shutdown_connections 2021-11-20T07:35:24.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:24.139+0000 7fbc4896e700 1 -- 172.21.15.71:0/3938331074 wait complete. 2021-11-20T07:35:24.543 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-11-20T07:35:24.544 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:35:24.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T07:35:24.613 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:35:24.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T07:35:24.678 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:35:24.679 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T07:35:24.709 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:35:24.710 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T07:35:24.775 INFO:tasks.cephadm:Deploying OSDs... 2021-11-20T07:35:24.775 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:35:24.776 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T07:35:24.792 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T07:35:24.793 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T07:35:24.849 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T07:35:24.849 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:24.849 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 118276 Links: 1 2021-11-20T07:35:24.849 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:24.850 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:24.850 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2021-11-20 07:34:55.216014273 +0000 2021-11-20T07:35:24.850 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2021-11-20 07:28:54.733291171 +0000 2021-11-20T07:35:24.850 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2021-11-20 07:28:54.733291171 +0000 2021-11-20T07:35:24.851 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2021-11-20T07:35:24.867 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2021-11-20T07:35:24.868 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2021-11-20T07:35:24.868 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00019848 s, 2.6 MB/s 2021-11-20T07:35:24.870 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T07:35:24.926 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T07:35:24.927 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:24.927 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 122945 Links: 1 2021-11-20T07:35:24.927 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:24.927 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:24.928 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2021-11-20 07:34:55.216014273 +0000 2021-11-20T07:35:24.928 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2021-11-20 07:28:55.074285438 +0000 2021-11-20T07:35:24.928 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2021-11-20 07:28:55.074285438 +0000 2021-11-20T07:35:24.928 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2021-11-20T07:35:24.937 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2021-11-20T07:35:24.938 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2021-11-20T07:35:24.938 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000212554 s, 2.4 MB/s 2021-11-20T07:35:24.940 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T07:35:24.995 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T07:35:24.996 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:24.996 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 113645 Links: 1 2021-11-20T07:35:24.997 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:24.997 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:24.997 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2021-11-20 07:34:55.216014273 +0000 2021-11-20T07:35:24.997 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2021-11-20 07:28:55.425279537 +0000 2021-11-20T07:35:24.998 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2021-11-20 07:28:55.425279537 +0000 2021-11-20T07:35:24.998 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2021-11-20T07:35:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2021-11-20T07:35:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2021-11-20T07:35:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000200959 s, 2.5 MB/s 2021-11-20T07:35:25.008 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T07:35:25.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:24 smithi112 conmon[36062]: audit 2021-11-20T07:35:24.137921+0000 mon.smithi071 2021-11-20T07:35:25.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:24 smithi112 conmon[36062]: (mon.0) 184 : audit [DBG] from='client.? 172.21.15.71:0/3938331074' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:25.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:24 smithi112 conmon[36062]: audit 2021-11-20T07:35:24.311576+0000 mon.smithi071 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:25.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:24 smithi112 conmon[36062]: cluster 2021-11-20T07:35:24.332345+0000 mgr.smithi071.ynqmwn (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:25.065 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T07:35:25.065 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:25.065 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 119537 Links: 1 2021-11-20T07:35:25.066 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:25.066 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:25.066 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2021-11-20 07:34:55.216014273 +0000 2021-11-20T07:35:25.066 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2021-11-20 07:28:55.776273634 +0000 2021-11-20T07:35:25.067 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2021-11-20 07:28:55.776273634 +0000 2021-11-20T07:35:25.067 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2021-11-20T07:35:25.073 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2021-11-20T07:35:25.074 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2021-11-20T07:35:25.074 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000184147 s, 2.8 MB/s 2021-11-20T07:35:25.076 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:35:25.077 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T07:35:25.093 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T07:35:25.094 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T07:35:25.148 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T07:35:25.149 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:25.149 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 116416 Links: 1 2021-11-20T07:35:25.149 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:25.149 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:25.150 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2021-11-20 07:35:19.549109539 +0000 2021-11-20T07:35:25.150 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2021-11-20 07:28:54.744932722 +0000 2021-11-20T07:35:25.150 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2021-11-20 07:28:54.744932722 +0000 2021-11-20T07:35:25.150 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2021-11-20T07:35:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2021-11-20T07:35:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2021-11-20T07:35:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000321434 s, 1.6 MB/s 2021-11-20T07:35:25.162 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T07:35:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T07:35:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:25.219 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 112373 Links: 1 2021-11-20T07:35:25.219 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:25.219 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:25.219 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2021-11-20 07:35:19.549109539 +0000 2021-11-20T07:35:25.220 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2021-11-20 07:28:55.076923942 +0000 2021-11-20T07:35:25.220 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2021-11-20 07:28:55.076923942 +0000 2021-11-20T07:35:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2021-11-20T07:35:25.228 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2021-11-20T07:35:25.229 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2021-11-20T07:35:25.229 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000226081 s, 2.3 MB/s 2021-11-20T07:35:25.231 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T07:35:25.286 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T07:35:25.287 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:25.287 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 111564 Links: 1 2021-11-20T07:35:25.287 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:25.288 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:25.288 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2021-11-20 07:35:19.550109513 +0000 2021-11-20T07:35:25.288 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2021-11-20 07:28:55.436914422 +0000 2021-11-20T07:35:25.288 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2021-11-20 07:28:55.436914422 +0000 2021-11-20T07:35:25.289 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2021-11-20T07:35:25.296 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2021-11-20T07:35:25.296 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2021-11-20T07:35:25.296 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000209138 s, 2.4 MB/s 2021-11-20T07:35:25.299 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T07:35:25.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:24 smithi071 conmon[33687]: audit 2021-11-20T07:35:24.137921+0000 mon.smithi071 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.71:0/3938331074' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:25.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:24 smithi071 conmon[33687]: audit 2021-11-20T07:35:24.311576+0000 mon.smithi071 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:25.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:24 smithi071 conmon[33687]: cluster 2021-11-20T07:35:24.332345+0000 mgr.smithi071.ynqmwn (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:25.359 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T07:35:25.360 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T07:35:25.360 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 114388 Links: 1 2021-11-20T07:35:25.360 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T07:35:25.360 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T07:35:25.361 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2021-11-20 07:35:19.550109513 +0000 2021-11-20T07:35:25.361 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2021-11-20 07:28:55.778905378 +0000 2021-11-20T07:35:25.361 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2021-11-20 07:28:55.778905378 +0000 2021-11-20T07:35:25.362 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2021-11-20T07:35:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2021-11-20T07:35:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2021-11-20T07:35:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000186775 s, 2.7 MB/s 2021-11-20T07:35:25.368 INFO:tasks.cephadm:Deploying osd.0 on smithi071 with /dev/vg_nvme/lv_4... 2021-11-20T07:35:25.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T07:35:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:26 smithi071 conmon[33687]: cluster 2021-11-20T07:35:26.332579+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:26 smithi071 conmon[33687]: .14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:26 smithi112 conmon[36062]: cluster 2021-11-20T07:35:26.332579+0000 mgr.smithi071.ynqmwn (mgr.14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:27.375 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T07:35:27.376 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T07:35:27.376 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2021-11-20T07:35:27.376 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2021-11-20T07:35:27.377 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113463 s, 924 MB/s 2021-11-20T07:35:27.377 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2021-11-20T07:35:27.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi071:vg_nvme/lv_4 2021-11-20T07:35:29.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: audit 2021-11-20T07:35:27.939862+0000 mon.smithi071 (mon.0) 186 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:29.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: cluster 2021-11-20T07:35:28.332819+0000 mgr.smithi071.ynqmwn (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:29.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: audit 2021-11-20T07:35:28.397861+0000 mon.smithi071 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:29.000 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: audit 2021-11-20T07:35:28.400734+0000 mon.smithi071 (mon.0) 188 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T07:35:29.001 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: audit 2021-11-20T07:35:28.402195+0000 mon.smithi071 (mon.0) 189 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T07:35:29.001 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:28 smithi071 conmon[33687]: audit 2021-11-20T07:35:28.402825+0000 mon.smithi071 (mon.0) 190 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:29.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: audit 2021-11-20T07:35:27.939862+0000 mon.smithi071 (mon.0) 186 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:29.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: cluster 2021-11-20T07:35:28.332819+0000 mgr.smithi071.ynqmwn (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:29.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: audit 2021-11-20T07:35:28.397861+0000 mon.smithi071 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:29.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: audit 2021-11-20T07:35:28.400734+0000 mon.smithi071 (mon.0) 188 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T07:35:29.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: audit 2021-11-20T07:35:28.402195+0000 mon.smithi071 (mon 2021-11-20T07:35:29.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: .0) 189 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T07:35:29.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:35:29.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:28 smithi112 conmon[36062]: :35:28.402825+0000 mon.smithi071 (mon.0) 190 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:29.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 -- 172.21.15.71:0/4056689488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 msgr2=0x7f72700f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:29.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 --2- 172.21.15.71:0/4056689488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f72700f6f10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7260004660 tx=0x7f7260009b30).stop 2021-11-20T07:35:29.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 -- 172.21.15.71:0/4056689488 shutdown_connections 2021-11-20T07:35:29.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 --2- 172.21.15.71:0/4056689488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 0x7f72700f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:29.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 --2- 172.21.15.71:0/4056689488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f72700f6f10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:29.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 -- 172.21.15.71:0/4056689488 >> 172.21.15.71:0/4056689488 conn(0x7f72700f0ba0 msgr2=0x7f72700f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:29.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 -- 172.21.15.71:0/4056689488 shutdown_connections 2021-11-20T07:35:29.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.429+0000 7f727481e700 1 -- 172.21.15.71:0/4056689488 wait complete. 2021-11-20T07:35:29.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 Processor -- start 2021-11-20T07:35:29.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 -- start start 2021-11-20T07:35:29.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:29.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 0x7f727010d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:29.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727010de30 con 0x7f7270060310 2021-11-20T07:35:29.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f727481e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727010df70 con 0x7f72700f7490 2021-11-20T07:35:29.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:29.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47290/0 (socket says 172.21.15.71:47290) 2021-11-20T07:35:29.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 -- 172.21.15.71:0/1246832883 learned_addr learned my addr 172.21.15.71:0/1246832883 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:35:29.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f7267fff700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 0x7f727010d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:29.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 -- 172.21.15.71:0/1246832883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 msgr2=0x7f727010d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:29.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 0x7f727010d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:29.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.430+0000 7f726f7fe700 1 -- 172.21.15.71:0/1246832883 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260005040 con 0x7f7270060310 2021-11-20T07:35:29.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f726f7fe700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7260007a40 tx=0x7f7260009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:29.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7260018070 con 0x7f7270060310 2021-11-20T07:35:29.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7260013410 con 0x7f7270060310 2021-11-20T07:35:29.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727010e8b0 con 0x7f7270060310 2021-11-20T07:35:29.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f726001c620 con 0x7f7270060310 2021-11-20T07:35:29.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.431+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727010ee10 con 0x7f7270060310 2021-11-20T07:35:29.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.432+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7270045ad0 con 0x7f7270060310 2021-11-20T07:35:29.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.432+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f726001d420 con 0x7f7270060310 2021-11-20T07:35:29.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.432+0000 7f726cff9700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 0x7f725c05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:29.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.432+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7260076d20 con 0x7f7270060310 2021-11-20T07:35:29.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.433+0000 7f7267fff700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 0x7f725c05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:29.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.433+0000 7f7267fff700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 0x7f725c05e7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f72580041d0 tx=0x7f7258009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:29.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.436+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f726004aa70 con 0x7f7270060310 2021-11-20T07:35:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:29.622+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7270105440 con 0x7f725c05c2f0 2021-11-20T07:35:29.946 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:28.400434 2021-11-20T07:35:29.947 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 47 : cephadm [INF] Reconfiguring mon.smithi071 (unknown last config time)... 2021-11-20T07:35:29.947 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:28.403544+0000 mgr.smithi071.ynqmwn (mgr.14162) 48 : cephadm [INF] Reconfiguring daemon mon.smithi071 on smithi071 2021-11-20T07:35:29.947 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.625130+0000 mon.smithi071 (mon.0) 191 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:35:29.947 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.626554+0000 mon.smithi071 (mon.0) 192 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:35:29.948 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.627031+0000 mon.smithi071 (mon.0) 193 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:29.948 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.737094+0000 mon.smithi071 (mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:29.948 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.737872+0000 mon.smithi071 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi071.ynqmwn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T07:35:29.948 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.738717+0000 mon.smithi071 (mon.0) 196 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T07:35:29.948 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:29 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.739319+0000 mon.smithi071 (mon.0) 197 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:30.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:28.400434+0000 mgr.smithi071.ynqmwn (mgr.14162) 47 : cephadm [INF] Reconfiguring mon.smithi071 (unknown last config time)... 2021-11-20T07:35:30.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:28.403544+0000 mgr.smithi071.ynqmwn (mgr.14162) 48 : cephadm [INF] Reconfiguring daemon mon.smithi071 on smithi071 2021-11-20T07:35:30.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07:35:29. 2021-11-20T07:35:30.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: 625130+0000 mon.smithi071 (mon.0) 191 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:35:30.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07: 2021-11-20T07:35:30.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: 35:29.626554+0000 mon.smithi071 (mon.0) 192 2021-11-20T07:35:30.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:35:30.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:35:30.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: :35:29.627031+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:30.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: 193 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:30.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:35:30.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: :35:29.737094+0000 mon.smithi071 (mon.0 2021-11-20T07:35:30.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: ) 194 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:30.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:35:30.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: T07:35:29.737872+0000 mon.smithi071 (mon 2021-11-20T07:35:30.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: .0) 195 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi071.ynqmwn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T07:35:30.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07: 2021-11-20T07:35:30.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: 35:29.738717+0000 mon.smithi071 (mon.0) 196 : audit 2021-11-20T07:35:30.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T07:35:30.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: audit 2021-11-20T07:35:29. 2021-11-20T07:35:30.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: 739319+0000 mon.smithi071 (mon.0) 197 : audit [DBG] 2021-11-20T07:35:30.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:29 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:31.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: audit 2021-11-20T07:35:29.624340+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:35:31.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: 14162) 49 : audit [DBG] from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:35:31.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:29.626376+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:35:31.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: ) 50 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:35:31.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:29. 2021-11-20T07:35:31.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: 737579+0000 mgr.smithi071.ynqmwn (mgr.14162) 51 : cephadm [INF] Reconfiguring mgr.smithi071.ynqmwn (unknown last config time)... 2021-11-20T07:35:31.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: cephadm 2021-11-20 2021-11-20T07:35:31.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: T07:35:29.739820+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:31.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: .14162) 52 : cephadm [INF] Reconfiguring daemon mgr.smithi071.ynqmwn on smithi071 2021-11-20T07:35:31.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:35:31.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: -11-20T07:35:30.333054+0000 mgr.smithi071.ynqmwn 2021-11-20T07:35:31.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: (mgr.14162) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:31.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:35:31.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: 11-20T07:35:31.182697+0000 2021-11-20T07:35:31.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: mon.smithi071 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:31.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:31 smithi112 conmon[36062]: 2021-11-20T07:35:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: audit 2021-11-20T07:35:29.624340+0000 mgr.smithi071.ynqmwn (mgr.14162) 49 : audit [DBG] from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:35:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:29.626376+0000 mgr.smithi071.ynqmwn (mgr.14162) 50 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:35:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:29.737579+0000 mgr.smithi071.ynqmwn (mgr.14162) 51 : cephadm [INF] Reconfiguring mgr.smithi071.ynqmwn (unknown last config time)... 2021-11-20T07:35:31.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:29.739820+0000 mgr.smithi071.ynqmwn (mgr.14162) 52 : cephadm [INF] Reconfiguring daemon mgr.smithi071.ynqmwn on smithi071 2021-11-20T07:35:31.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: cluster 2021-11-20T07:35:30.333054+0000 mgr.smithi071.ynqmwn (mgr.14162) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:31.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:31 smithi071 conmon[33687]: audit 2021-11-20T07:35:31.182697+0000 mon.smithi071 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:32.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:31.183202+0000 mgr.smithi071.ynqmwn (mgr.14162) 54 : cephadm [INF] Reconfiguring alertmanager.smithi071 (dependencies changed)... 2021-11-20T07:35:32.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: cephadm 2021-11-20 2021-11-20T07:35:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: T07:35:31.185616+0000 mgr.smithi071.ynqmwn (mgr.14162) 55 : cephadm [INF] Reconfiguring daemon alertmanager.smithi071 on smithi071 2021-11-20T07:35:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: audit 2021-11-20T07:35:32.127484+0000 mon.smithi071 (mon.0) 199 : audit [INF] from='client.? 172.21.15.71:0/870437093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06586a57-2910-4cab-8442-30fd5e92b95d"}]: dispatch 2021-11-20T07:35:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: audit 2021-11-20T07:35:32.131687+0000 mon.smithi071 (mon.0 2021-11-20T07:35:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: ) 200 : audit [INF] from='client.? 172.21.15.71:0/870437093' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06586a57-2910-4cab-8442-30fd5e92b95d"}]': finished 2021-11-20T07:35:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:35:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: T07:35:32.131728+0000 mon.smithi071 (mon. 2021-11-20T07:35:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: 0) 201 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T07:35:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:35:32.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: 20T07:35:32.131843+0000 mon.smithi071 ( 2021-11-20T07:35:32.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:32 smithi112 conmon[36062]: mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:31.183202+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: .14162) 54 : cephadm [INF] Reconfiguring alertmanager.smithi071 (dependencies changed)... 2021-11-20T07:35:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:31.185616+0000 mgr.smithi071.ynqmwn (mgr.14162) 55 : cephadm [INF] Reconfiguring daemon alertmanager.smithi071 on smithi071 2021-11-20T07:35:32.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: audit 2021-11-20T07:35:32.127484+0000 mon.smithi071 (mon.0) 199 : audit [INF] from='client.? 172.21.15.71:0/870437093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06586a57-2910-4cab-8442-30fd5e92b95d"}]: dispatch 2021-11-20T07:35:32.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: audit 2021-11-20T07:35:32.131687+0000 mon.smithi071 (mon.0) 200 : audit [INF] from='client.? 172.21.15.71:0/870437093' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06586a57-2910-4cab-8442-30fd5e92b95d"}]': finished 2021-11-20T07:35:32.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: cluster 2021-11-20T07:35:32.131728+0000 mon.smithi071 (mon.0) 201 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T07:35:32.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:32 smithi071 conmon[33687]: audit 2021-11-20T07:35:32.131843+0000 mon.smithi071 (mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:33 smithi112 conmon[36062]: cluster 2021-11-20T07:35:32.333262+0000 mgr.smithi071.ynqmwn (mgr.14162) 56 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:33 smithi112 conmon[36062]: audit 2021-11-20T07:35:32.542816+0000 mon.smithi071 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.71:0/293062084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:35:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:33 smithi071 conmon[33687]: cluster 2021-11-20T07:35:32.333262+0000 mgr.smithi071.ynqmwn (mgr.14162) 56 : cluster 2021-11-20T07:35:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:33 smithi071 conmon[33687]: [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:33 smithi071 conmon[33687]: audit 2021-11-20T07:35:32.542816+0000 mon.smithi071 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.71:0/293062084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:35:35.772 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:35 smithi071 conmon[33687]: cluster 2021-11-20T07:35:34.333510+0000 mgr.smithi071.ynqmwn (mgr.14162) 57 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:35 smithi112 conmon[36062]: cluster 2021-11-20T07:35:34.333510+0000 mgr.smithi071.ynqmwn (mgr.14162) 57 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:37.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:37 smithi071 conmon[33687]: cluster 2021-11-20T07:35:36.333780+0000 mgr.smithi071.ynqmwn (mgr.14162) 58 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:37 smithi112 conmon[36062]: cluster 2021-11-20T07:35:36.333780+0000 mgr.smithi071.ynqmwn (mgr.14162) 58 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:39.525 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:39 smithi071 conmon[33687]: cluster 2021-11-20T07:35:38.334053+0000 mgr.smithi071.ynqmwn 2021-11-20T07:35:39.526 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:39 smithi071 conmon[33687]: (mgr.14162) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:39.526 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:39 smithi071 conmon[33687]: audit 2021-11-20T07:35:39.302872+0000 mon.smithi071 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:39.527 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:39 smithi071 conmon[33687]: audit 2021-11-20T07:35:39.303645+0000 mon.smithi071 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi071", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:35:39.527 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:39 smithi071 conmon[33687]: audit 2021-11-20T07:35:39.304272+0000 mon.smithi071 (mon.0) 206 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: cluster 2021-11-20T07:35:38.334053+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: .14162) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: audit 2021-11-20T07:35:39.302872+0000 mon.smithi071 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: audit 2021-11-20T07:35:39.303645+0000 2021-11-20T07:35:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: mon.smithi071 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi071", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:35:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: 2021-11-20T07:35:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: audit 2021-11-20T07:35:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: 2021-11-20T07:35:39.304272+0000 mon.smithi071 ( 2021-11-20T07:35:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:39 smithi112 conmon[36062]: mon.0) 206 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:40.522 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:40 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:39.303351+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:40.523 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:40 smithi071 conmon[33687]: mgr.14162) 60 : cephadm [INF] Reconfiguring crash.smithi071 (monmap changed)... 2021-11-20T07:35:40.523 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:40 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:39.304681+0000 mgr.smithi071.ynqmwn (mgr.14162) 61 : cephadm [INF] Reconfiguring daemon crash.smithi071 on smithi071 2021-11-20T07:35:40.523 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:40 smithi071 conmon[33687]: audit 2021-11-20T07:35:40.284416+0000 mon.smithi071 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T07:35:40.524 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:40 smithi071 conmon[33687]: audit 2021-11-20T07:35:40.285015+0000 mon.smithi071 (mon.0) 208 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:39.303351+0000 mgr.smithi071.ynqmwn (mgr.14162) 60 : cephadm [INF] Reconfiguring crash.smithi071 (monmap changed)... 2021-11-20T07:35:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:39.304681+0000 mgr.smithi071.ynqmwn (mgr.14162) 61 : cephadm [INF] Reconfiguring daemon crash.smithi071 on smithi071 2021-11-20T07:35:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:35:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: 20T07:35:40.284416+0000 mon.smithi071 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T07:35:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:35:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: 11-20T07:35:40.285015+0000 2021-11-20T07:35:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: mon.smithi071 (mon.0) 208 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:40 smithi112 conmon[36062]: 2021-11-20T07:35:41.501 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:41 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:40.285477+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:41.501 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:41 smithi071 conmon[33687]: .14162) 62 : cephadm [INF] Deploying daemon osd.0 on smithi071 2021-11-20T07:35:41.501 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:41 smithi071 conmon[33687]: cluster 2021-11-20T07:35:40.334312+0000 mgr.smithi071.ynqmwn (mgr.14162) 63 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:41.502 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:41 smithi071 conmon[33687]: audit 2021-11-20T07:35: 2021-11-20T07:35:41.502 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:41 smithi071 conmon[33687]: 41.199334+0000 mon.smithi071 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:41 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:40.285477+0000 mgr.smithi071.ynqmwn (mgr.14162) 62 : cephadm [INF] Deploying daemon osd.0 on smithi071 2021-11-20T07:35:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:41 smithi112 conmon[36062]: cluster 2021-11-20T07:35:40.334312+0000 mgr.smithi071.ynqmwn (mgr.14162) 63 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:41 smithi112 conmon[36062]: audit 2021 2021-11-20T07:35:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:41 smithi112 conmon[36062]: -11-20T07:35:41.199334+0000 mon.smithi071 (mon.0) 209 : audit [INF] 2021-11-20T07:35:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:41 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:42.610 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:42 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:41.199885+0000 mgr.smithi071.ynqmwn (mgr.14162) 64 : cephadm [INF] Reconfiguring grafana.smithi071 (dependencies changed)... 2021-11-20T07:35:42.611 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:42 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:41.204829+0000 mgr.smithi071.ynqmwn (mgr.14162) 65 : cephadm [INF] Reconfiguring daemon grafana.smithi071 on smithi071 2021-11-20T07:35:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:42 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:41.199885+0000 mgr.smithi071.ynqmwn (mgr.14162) 64 : cephadm [INF] Reconfiguring grafana.smithi071 (dependencies changed)... 2021-11-20T07:35:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:42 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:41.204829+0000 mgr.smithi071.ynqmwn (mgr.14162) 65 : cephadm [INF] Reconfiguring daemon grafana.smithi071 on smithi071 2021-11-20T07:35:43.631 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:43 smithi071 conmon[33687]: cluster 2021-11-20T07:35:42.334556+0000 mgr.smithi071.ynqmwn (mgr.14162) 66 : cluster 2021-11-20T07:35:43.631 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:43 smithi071 conmon[33687]: [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:43 smithi112 conmon[36062]: cluster 2021-11-20T07:35:42.334556+0000 mgr.smithi071.ynqmwn (mgr.14162) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:43.910 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 0 on host 'smithi071' 2021-11-20T07:35:43.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.907+0000 7f726cff9700 1 -- 172.21.15.71:0/1246832883 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7270105440 con 0x7f725c05c2f0 2021-11-20T07:35:43.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 msgr2=0x7f725c05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:43.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 0x7f725c05e7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f72580041d0 tx=0x7f7258009040).stop 2021-11-20T07:35:43.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 msgr2=0x7f727010d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:43.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7260007a40 tx=0x7f7260009b30).stop 2021-11-20T07:35:43.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 shutdown_connections 2021-11-20T07:35:43.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f725c05c2f0 0x7f725c05e7b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:43.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7270060310 0x7f727010d390 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:43.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 --2- 172.21.15.71:0/1246832883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72700f7490 0x7f727010d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:43.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 >> 172.21.15.71:0/1246832883 conn(0x7f72700f0ba0 msgr2=0x7f72700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:43.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 shutdown_connections 2021-11-20T07:35:43.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:43.909+0000 7f727481e700 1 -- 172.21.15.71:0/1246832883 wait complete. 2021-11-20T07:35:44.334 DEBUG:teuthology.orchestra.run.smithi071:osd.0> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.0.service 2021-11-20T07:35:44.337 INFO:tasks.cephadm:Deploying osd.1 on smithi071 with /dev/vg_nvme/lv_3... 2021-11-20T07:35:44.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T07:35:44.366 INFO:journalctl@ceph.osd.0.smithi071.stdout:-- Logs begin at Sat 2021-11-20 07:22:38 UTC. -- 2021-11-20T07:35:45.121 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:44 smithi071 conmon[33687]: audit 2021-11-20T07:35:43.904060+0000 mon.smithi071 2021-11-20T07:35:45.121 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:44 smithi071 conmon[33687]: (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:45.121 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:44 smithi071 conmon[33687]: audit 2021-11-20T07:35:43.907701+0000 mon.smithi071 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:45.122 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:44 smithi071 conmon[33687]: cluster 2021-11-20T07:35:44.334961+0000 mgr.smithi071.ynqmwn (mgr.14162) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:45.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:44 smithi112 conmon[36062]: audit 2021-11-20T07:35:43.904060+0000 mon.smithi071 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:45.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:44 smithi112 conmon[36062]: audit 2021-11-20T07:35:43.907701+0000 mon.smithi071 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:45.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:44 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:35:45.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:44 smithi112 conmon[36062]: 35:44.334961+0000 mgr.smithi071.ynqmwn (mgr.14162) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:46.449 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:46 smithi071 conmon[33687]: cluster 2021-11-20T07:35: 2021-11-20T07:35:46.450 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:46 smithi071 conmon[33687]: 46.335239+0000 mgr.smithi071.ynqmwn (mgr.14162) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:46.761 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T07:35:46.762 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T07:35:46.762 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2021-11-20T07:35:46.762 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2021-11-20T07:35:46.763 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121832 s, 861 MB/s 2021-11-20T07:35:46.763 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2021-11-20T07:35:46.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:46 smithi112 conmon[36062]: cluster 2021-11-20T07:35:46.335239+0000 mgr.smithi071.ynqmwn (mgr.14162) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:47.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi071:vg_nvme/lv_3 2021-11-20T07:35:47.795 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:35:47 smithi071 conmon[50727]: debug 2021-11-20T07:35:47.780+0000 7fac477fb080 -1 osd.0 0 log_to_monitors {default=true} 2021-11-20T07:35:49.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: audit 2021-11-20T07:35:47.694038+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:49.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: 212 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:49.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:47.694810+0000 mgr.smithi071.ynqmwn (mgr.14162) 69 : cephadm [INF] Reconfiguring prometheus.smithi071 (dependencies changed)... 2021-11-20T07:35:49.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:47.696791+0000 mgr.smithi071.ynqmwn (mgr.14162) 70 : cephadm [INF] 2021-11-20T07:35:49.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: Reconfiguring daemon prometheus.smithi071 on smithi071 2021-11-20T07:35:49.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: audit 2021-11-20T07:35 2021-11-20T07:35:49.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: :47.783657+0000 mon.smithi071 (mon.0) 213 : 2021-11-20T07:35:49.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T07:35:49.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:35:49.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: :35:48.335641+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:35:49.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:48 smithi112 conmon[36062]: 71 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:49.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:35:49.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: 35:47.694038+0000 mon.smithi071 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:49.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:47.694810+0000 mgr.smithi071.ynqmwn (mgr.14162) 69 : cephadm [INF] Reconfiguring prometheus.smithi071 (dependencies changed)... 2021-11-20T07:35:49.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:47.696791+0000 mgr.smithi071.ynqmwn (mgr.14162) 70 : cephadm [INF] Reconfiguring daemon prometheus.smithi071 on smithi071 2021-11-20T07:35:49.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: audit 2021-11-20T07:35:47.783657+0000 mon.smithi071 (mon.0) 213 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T07:35:49.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:48 smithi071 conmon[33687]: cluster 2021-11-20T07:35:48.335641+0000 mgr.smithi071.ynqmwn (mgr.14162) 71 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:49.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 -- 172.21.15.71:0/3135049709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f66c0 msgr2=0x7f16580f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:49.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 --2- 172.21.15.71:0/3135049709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f66c0 0x7f16580f6ae0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f1648004660 tx=0x7f1648009b30).stop 2021-11-20T07:35:49.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 -- 172.21.15.71:0/3135049709 shutdown_connections 2021-11-20T07:35:49.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 --2- 172.21.15.71:0/3135049709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f7960 0x7f16580f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:35:49.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 --2- 172.21.15.71:0/3135049709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f66c0 0x7f16580f6ae0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 -- 172.21.15.71:0/3135049709 >> 172.21.15.71:0/3135049709 conn(0x7f16580f1e20 msgr2=0x7f16580f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:35:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 -- 172.21.15.71:0/3135049709 shutdown_connections 2021-11-20T07:35:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 -- 172.21.15.71:0/3135049709 wait complete. 2021-11-20T07:35:49.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.172+0000 7f165dfef700 1 Processor -- start 2021-11-20T07:35:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f165dfef700 1 -- start start 2021-11-20T07:35:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f165dfef700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f66c0 0x7f1658106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f165dfef700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f165dfef700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1658107ae0 con 0x7f16580f7960 2021-11-20T07:35:49.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f165dfef700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1658108390 con 0x7f16580f66c0 2021-11-20T07:35:49.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:49.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47334/0 (socket says 172.21.15.71:47334) 2021-11-20T07:35:49.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 -- 172.21.15.71:0/2051232401 learned_addr learned my addr 172.21.15.71:0/2051232401 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:35:49.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f16577fe700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f66c0 0x7f1658106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:49.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 -- 172.21.15.71:0/2051232401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f66c0 msgr2=0x7f1658106fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:35:49.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f66c0 0x7f1658106fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:35:49.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 -- 172.21.15.71:0/2051232401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648005040 con 0x7f16580f7960 2021-11-20T07:35:49.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.173+0000 7f1656ffd700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f164c00b410 tx=0x7f164c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:49.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.174+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c0125f0 con 0x7f16580f7960 2021-11-20T07:35:49.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.174+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f164c00be30 con 0x7f16580f7960 2021-11-20T07:35:49.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.174+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1658108670 con 0x7f16580f7960 2021-11-20T07:35:49.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.174+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c002a20 con 0x7f16580f7960 2021-11-20T07:35:49.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.174+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1658108c30 con 0x7f16580f7960 2021-11-20T07:35:49.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.175+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f164c00f030 con 0x7f16580f7960 2021-11-20T07:35:49.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.175+0000 7f163ffff700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 0x7f164005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:35:49.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.175+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1476+0+0 (secure 0 0 0) 0x7f164c07a7b0 con 0x7f16580f7960 2021-11-20T07:35:49.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.175+0000 7f16577fe700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 0x7f164005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:35:49.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.175+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1658045ad0 con 0x7f16580f7960 2021-11-20T07:35:49.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.176+0000 7f16577fe700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 0x7f164005e770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f1648004630 tx=0x7f1648009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:35:49.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.180+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f164c07b060 con 0x7f16580f7960 2021-11-20T07:35:49.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:35:49.363+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f16580fefa0 con 0x7f164005c2b0 2021-11-20T07:35:50.010 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[50727]: debug 2021-11-20T07:35:49.705+0000 7fac3dd6c700 -1 osd.0 0 waiting for initial osdmap 2021-11-20T07:35:50.011 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[50727]: debug 2021-11-20T07:35:49.713+0000 7fac384e4700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:35:50.012 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021-11-20T07:35:48.700843+0000 2021-11-20T07:35:50.012 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: mon.smithi071 (mon.0) 214 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T07:35:50.016 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: cluster 2021-11-20T07:35:50.016 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 2021-11- 2021-11-20T07:35:50.016 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 20T07:35:48.700913+0000 mon.smithi071 (mon 2021-11-20T07:35:50.016 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: .0) 215 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T07:35:50.017 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021-11-20T07:35 2021-11-20T07:35:50.017 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: :48.701050+0000 mon.smithi071 (mon.0) 216 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:50.017 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021-11-20T07:35:48.701539+0000 mon.smithi071 ( 2021-11-20T07:35:50.017 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: mon.0) 217 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:35:50.018 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:50.018 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: -20T07:35:49.364424+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:50.018 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: .14162) 72 : audit [DBG] from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:35:50.018 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:35:50.019 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 20T07:35:49.365174+0000 mon.smithi071 (mon 2021-11-20T07:35:50.019 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: .0) 218 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:35:50.019 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: cephadm 2021-11 2021-11-20T07:35:50.019 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: -20T07:35:49.366184+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:50.019 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: mgr.14162) 73 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:35:50.020 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:35:50.020 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 11-20T07:35:49.366370+0000 mon.smithi071 2021-11-20T07:35:50.020 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:35:50.020 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:35:50.021 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 11-20T07:35:49.366940+0000 2021-11-20T07:35:50.021 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: mon.smithi071 (mon.0) 220 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:50.021 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:49 smithi071 conmon[33687]: 2021-11-20T07:35:50.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021-11-20T07:35:48.700843+0000 mon.smithi071 (mon. 2021-11-20T07:35:50.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 0) 214 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T07:35:50.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: cluster 2021-11-20T07:35:48.700913+0000 mon.smithi071 (mon.0) 215 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T07:35:50.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021-11-20T07:35:48.701050+0000 mon.smithi071 (mon. 2021-11-20T07:35:50.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 0) 216 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:50.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:35:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: :35:48.701539+0000 mon.smithi071 (mon.0) 217 2021-11-20T07:35:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:35:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:35:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 07:35:49.364424+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:35:50.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: ) 72 : audit [DBG] from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:35:50.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:35:50.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: T07:35:49.365174+0000 mon.smithi071 (mon. 2021-11-20T07:35:50.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 0) 218 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:35:50.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: cephadm 2021-11 2021-11-20T07:35:50.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: -20T07:35:49.366184+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:50.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: mgr.14162) 73 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:35:50.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:35:50.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 11-20T07:35:49.366370+0000 mon.smithi071 2021-11-20T07:35:50.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:35:50.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: audit 2021 2021-11-20T07:35:50.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: -11-20T07:35:49.366940+0000 2021-11-20T07:35:50.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: mon.smithi071 (mon.0) 220 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:50.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:49 smithi112 conmon[36062]: 2021-11-20T07:35:50.718 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: audit 2021-11-20T07:35:49.702684+0000 mon.smithi071 2021-11-20T07:35:50.718 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: (mon.0) 221 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:35:50.718 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: cluster 2021-11-20T07:35:49.702743+0000 mon.smithi071 (mon.0) 222 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T07:35:50.993 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: audit 2021-11-20T07:35:49.703082+0000 mon.smithi071 (mon.0) 223 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:50.997 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: audit 2021-11-20T07:35:49.709002+0000 mon.smithi071 (mon.0) 224 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:50.997 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:50 smithi071 conmon[33687]: cluster 2021-11-20T07:35:50.335973+0000 mgr.smithi071.ynqmwn (mgr.14162) 74 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:51.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: audit 2021-11-20T07:35:49.702684+0000 mon.smithi071 (mon.0 2021-11-20T07:35:51.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: ) 221 : audit [INF] from='osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:35:51.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: cluster 2021-11-20T07:35:49.702743+0000 mon.smithi071 (mon.0) 222 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T07:35:51.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: audit 2021-11-20T07:35:49.703082+0000 2021-11-20T07:35:51.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: mon.smithi071 (mon.0) 223 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:51.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:35:51.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: -20T07:35:49.709002+0000 mon.smithi071 (mon 2021-11-20T07:35:51.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: .0) 224 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:51.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: cluster 2021-11-20T07:35:50.335973+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:51.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:50 smithi112 conmon[36062]: mgr.14162) 74 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T07:35:51.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: cluster 2021-11-20T07:35:48.763645+0000 osd.0 ( 2021-11-20T07:35:51.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:35:51.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: cluster 2021-11-20T07:35:48.763752+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:35:51.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit 2021-11-20T07:35:50.705790+0000 mon.smithi071 (mon.0) 225 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:51.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: cluster 2021-11-20T07:35:50.713890+0000 mon.smithi071 (mon.0) 226 : cluster [INF] osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] boot 2021-11-20T07:35:51.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: cluster 2021-11-20T07:35:51.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: 2021-11-20T07:35:50.713924+0000 mon.smithi071 (mon.0) 227 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T07:35:51.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:51.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: -20T07:35:50.714989+0000 mon.smithi071 (mon.0) 228 : 2021-11-20T07:35:51.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:51.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:35:51.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: 35:51.596387+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:51.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: 229 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:51.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:35:51.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: 20T07:35:51.597146+0000 mon.smithi071 (mon 2021-11-20T07:35:51.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: .0) 230 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:35:51.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: audit 2021 2021-11-20T07:35:51.829 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: -11-20T07:35:51.597730+0000 2021-11-20T07:35:51.829 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: mon.smithi071 (mon.0) 231 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:51.829 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:51 smithi071 conmon[33687]: 2021-11-20T07:35:52.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: cluster 2021-11-20T07:35:48.763645+0000 osd.0 (osd.0) 1 : cluster 2021-11-20T07:35:52.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: [DBG] purged_snaps scrub starts 2021-11-20T07:35:52.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: cluster 2021-11-20T07:35:48.763752+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:35:52.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:35:52.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: -20T07:35:50.705790+0000 mon.smithi071 (mon.0) 225 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:52.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:35:52.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: :35:50.713890+0000 mon.smithi071 (mon.0) 226 : cluster [INF] osd.0 [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] boot 2021-11-20T07:35:52.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: cluster 2021-11-20T07:35:50.713924+0000 mon.smithi071 ( 2021-11-20T07:35:52.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: mon.0) 227 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T07:35:52.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: 2021-11-20T07:35:50.714989+0000 mon.smithi071 (mon.0) 228 : audit [DBG] 2021-11-20T07:35:52.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T07:35:52.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: 2021-11-20T07:35:51.596387+0000 mon.smithi071 (mon.0) 229 : audit [INF] 2021-11-20T07:35:52.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:52.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: audit 2021 2021-11-20T07:35:52.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: -11-20T07:35:51.597146+0000 mon.smithi071 (mon.0) 230 : audit [INF] 2021-11-20T07:35:52.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T07:35:52.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: 2021-11-20T07:35:51.597730+0000 mon.smithi071 (mon.0) 231 : audit [DBG] 2021-11-20T07:35:52.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:51 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:53.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:51.596898+0000 mgr.smithi071.ynqmwn (mgr.14162) 75 : cephadm [INF] Reconfiguring crash.smithi112 (monmap changed)... 2021-11-20T07:35:53.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:51.598134+0000 mgr.smithi071.ynqmwn (mgr.14162) 76 : cephadm [INF] Reconfiguring daemon crash.smithi112 on smithi112 2021-11-20T07:35:53.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: cluster 2021-11-20T07:35:51.719897+0000 mon.smithi071 (mon.0) 232 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T07:35:53.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: cluster 2021-11-20T07:35:52.336231+0000 mgr.smithi071.ynqmwn (mgr.14162) 77 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:53.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.487299+0000 mon.smithi071 (mon.0) 233 : audit [INF] from='client.? 172.21.15.71:0/2742600639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e5e3ee-bd5a-4777-94b2-d475e3515783"}]: dispatch 2021-11-20T07:35:53.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.490649+0000 mon.smithi071 (mon.0) 234 : audit [INF] from='client.? 172.21.15.71:0/2742600639' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e5e3ee-bd5a-4777-94b2-d475e3515783"}]': finished 2021-11-20T07:35:53.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: cluster 2021-11-20T07:35:52.490690+0000 mon.smithi071 (mon.0) 235 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T07:35:53.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:52 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.490783+0000 mon.smithi071 (mon.0) 236 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:35:53.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:51.596898+0000 mgr.smithi071.ynqmwn 2021-11-20T07:35:53.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: (mgr.14162) 75 : cephadm [INF] Reconfiguring crash.smithi112 (monmap changed)... 2021-11-20T07:35:53.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:51.598134+0000 mgr.smithi071.ynqmwn (mgr.14162) 76 : cephadm [INF] Reconfiguring daemon crash.smithi112 on smithi112 2021-11-20T07:35:53.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: cluster 2021-11-20T07:35:51.719897+0000 mon.smithi071 (mon.0) 232 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T07:35:53.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: cluster 2021-11-20T07:35:52.336231+0000 mgr.smithi071.ynqmwn (mgr.14162) 77 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:53.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.487299+0000 mon.smithi071 (mon.0) 233 : audit [INF] from='client.? 172.21.15.71:0/2742600639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e5e3ee-bd5a-4777-94b2-d475e3515783"}]: dispatch 2021-11-20T07:35:53.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.490649+0000 mon.smithi071 (mon.0) 234 : audit [INF] from='client.? 172.21.15.71:0/2742600639' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e5e3ee-bd5a-4777-94b2-d475e3515783"}]': finished 2021-11-20T07:35:53.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: cluster 2021-11-20T07:35:52.490690+0000 mon.smithi071 (mon.0) 235 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T07:35:53.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:52 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.490783+0000 mon.smithi071 (mon.0) 236 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:35:54.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.776102+0000 mon.smithi071 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:54.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:52.776551+0000 mgr.smithi071.ynqmwn (mgr.14162) 78 : cephadm [INF] Reconfiguring mgr.smithi112.qaikrr (monmap changed)... 2021-11-20T07:35:54.045 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.776804+0000 mon.smithi071 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.qaikrr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T07:35:54.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.777534+0000 mon.smithi071 (mon.0) 239 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T07:35:54.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.778069+0000 mon.smithi071 (mon.0) 240 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:54.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:52.778530+0000 mgr.smithi071.ynqmwn (mgr.14162) 79 : cephadm [INF] Reconfiguring daemon mgr.smithi112.qaikrr on smithi112 2021-11-20T07:35:54.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:53 smithi112 conmon[36062]: audit 2021-11-20T07:35:52.912023+0000 mon.smithi071 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.71:0/2818798616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:35:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.776102+0000 mon.smithi071 (mon 2021-11-20T07:35:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: .0) 237 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:54.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:52.776551+0000 mgr.smithi071.ynqmwn (mgr.14162) 78 : cephadm [INF] Reconfiguring mgr.smithi112.qaikrr (monmap changed)... 2021-11-20T07:35:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.776804+0000 mon.smithi071 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.qaikrr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T07:35:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: audit 2021-11-20T07:35:52.777534+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: 239 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T07:35:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:35:54.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: -20T07:35:52.778069+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:54.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: 240 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:54.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: cephadm 2021-11 2021-11-20T07:35:54.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: -20T07:35:52.778530+0000 mgr.smithi071.ynqmwn 2021-11-20T07:35:54.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: (mgr.14162) 79 : cephadm [INF] Reconfiguring daemon mgr.smithi112.qaikrr on smithi112 2021-11-20T07:35:54.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: audit 2021 2021-11-20T07:35:54.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: -11-20T07:35:52.912023+0000 2021-11-20T07:35:54.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: mon.smithi071 (mon.0) 241 : audit [DBG] 2021-11-20T07:35:54.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:53 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/2818798616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:35:55.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: audit 2021-11-20T07:35:53.945799+0000 mon.smithi071 (mon.0) 2021-11-20T07:35:55.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: 242 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:55.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:53.946246+0000 mgr.smithi071.ynqmwn (mgr.14162) 80 : cephadm [INF] Reconfiguring mon.smithi112 (monmap changed)... 2021-11-20T07:35:55.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: audit 2021-11-20T07:35:53.946447+0000 mon.smithi071 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T07:35:55.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: audit 2021-11-20T07:35:53.946986+0000 mon.smithi071 (mon.0) 244 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T07:35:55.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: audit 2021-11-20T07:35:53.947506+0000 mon.smithi071 (mon.0) 245 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:55.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:53.947943+0000 mgr.smithi071.ynqmwn (mgr.14162) 81 : cephadm [INF] Reconfiguring daemon mon.smithi112 on smithi112 2021-11-20T07:35:55.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:54 smithi112 conmon[36062]: cluster 2021-11-20T07:35:54.336602+0000 mgr.smithi071.ynqmwn (mgr.14162) 82 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:55.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: audit 2021-11-20T07:35:53.945799+0000 mon.smithi071 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:55.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: cephadm 2021-11-20T07 2021-11-20T07:35:55.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: :35:53.946246+0000 mgr.smithi071.ynqmwn (mgr.14162) 80 : cephadm [INF] Reconfiguring mon.smithi112 (monmap changed)... 2021-11-20T07:35:55.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: audit 2021-11-20T07:35:53.946447+0000 mon.smithi071 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T07:35:55.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: audit 2021-11-20T07:35:53. 2021-11-20T07:35:55.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: 946986+0000 mon.smithi071 (mon.0) 244 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T07:35:55.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: audit 2021-11-20T07:35:55.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: 2021-11-20T07 2021-11-20T07:35:55.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: :35:53.947506+0000 mon.smithi071 (mon.0 2021-11-20T07:35:55.322 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: ) 245 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:55.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: cephadm 2021-11-20T 2021-11-20T07:35:55.323 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: 07:35:53.947943+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:55.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: .14162) 81 : cephadm [INF] Reconfiguring daemon mon.smithi112 on smithi112 2021-11-20T07:35:55.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:35:55.324 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: 11-20T07:35:54.336602+0000 2021-11-20T07:35:55.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 82 : cluster [DBG] 2021-11-20T07:35:55.325 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:54 smithi071 conmon[33687]: pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:56.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:56 smithi112 conmon[36062]: audit 2021-11-20T07:35:55.010995+0000 mon.smithi071 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:56.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:56 smithi112 conmon[36062]: audit 2021-11-20T07:35:55.013355+0000 mon.smithi071 (mon.0) 247 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T07:35:56.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:56 smithi112 conmon[36062]: audit 2021-11-20T07:35:55.013715+0000 mgr.smithi071.ynqmwn (mgr.14162) 83 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T07:35:56.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:56 smithi071 conmon[33687]: audit 2021-11-20T07:35:55.010995+0000 mon.smithi071 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:35:56.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:56 smithi071 conmon[33687]: audit 2021-11-20T07:35:55.013355+0000 mon.smithi071 (mon.0) 247 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T07:35:56.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:56 smithi071 conmon[33687]: audit 2021-11-20T07:35 2021-11-20T07:35:56.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:56 smithi071 conmon[33687]: :55.013715+0000 mgr.smithi071.ynqmwn (mgr.14162) 83 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T07:35:57.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:57 smithi112 conmon[36062]: cluster 2021-11-20T07:35:56.336931+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:35:57.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:57 smithi112 conmon[36062]: .14162) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:57.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:57 smithi071 conmon[33687]: cluster 2021-11-20T07:35:56.336931+0000 mgr.smithi071.ynqmwn (mgr.14162) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:58.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:58 smithi071 conmon[33687]: cluster 2021-11-20T07:35:58.337232+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:58.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:58 smithi071 conmon[33687]: mgr.14162) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:58 smithi112 conmon[36062]: cluster 2021-11-20T07:35:58.337232+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:35:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:58 smithi112 conmon[36062]: mgr.14162) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:35:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:59 smithi071 conmon[33687]: audit 2021-11-20T07:35:58.665285+0000 mon.smithi071 (mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T07:35:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:59 smithi071 conmon[33687]: audit 2021-11-20T07:35:58.665851+0000 mon.smithi071 (mon.0) 249 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:35:59 smithi071 conmon[33687]: cephadm 2021-11-20T07:35:58.666316+0000 mgr.smithi071.ynqmwn (mgr.14162) 86 : cephadm [INF] Deploying daemon osd.1 on smithi071 2021-11-20T07:35:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:59 smithi112 conmon[36062]: audit 2021-11-20T07:35:58.665285+0000 mon.smithi071 ( 2021-11-20T07:35:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:59 smithi112 conmon[36062]: mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T07:35:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:59 smithi112 conmon[36062]: audit 2021-11-20T07:35:58.665851+0000 mon.smithi071 (mon.0) 249 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:35:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:35:59 smithi112 conmon[36062]: cephadm 2021-11-20T07:35:58.666316+0000 mgr.smithi071.ynqmwn (mgr.14162) 86 : cephadm [INF] Deploying daemon osd.1 on smithi071 2021-11-20T07:36:01.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:01 smithi071 conmon[33687]: cluster 2021-11-20T07:36:00.337594+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:36:01.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:01 smithi071 conmon[33687]: .14162) 87 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:01.569 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 1 on host 'smithi071' 2021-11-20T07:36:01.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.565+0000 7f163ffff700 1 -- 172.21.15.71:0/2051232401 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f16580fefa0 con 0x7f164005c2b0 2021-11-20T07:36:01.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 msgr2=0x7f164005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:01.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 0x7f164005e770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f1648004630 tx=0x7f1648009b30).stop 2021-11-20T07:36:01.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 msgr2=0x7f16581074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:01.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f164c00b410 tx=0x7f164c006ef0).stop 2021-11-20T07:36:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 shutdown_connections 2021-11-20T07:36:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16580f66c0 0x7f1658106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f164005c2b0 0x7f164005e770 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 --2- 172.21.15.71:0/2051232401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16580f7960 0x7f16581074a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:01.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 >> 172.21.15.71:0/2051232401 conn(0x7f16580f1e20 msgr2=0x7f16580fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:01.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 shutdown_connections 2021-11-20T07:36:01.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:01.567+0000 7f165dfef700 1 -- 172.21.15.71:0/2051232401 wait complete. 2021-11-20T07:36:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:01 smithi112 conmon[36062]: cluster 2021-11-20T07:36:00.337594+0000 mgr.smithi071.ynqmwn (mgr.14162) 87 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:01.890 DEBUG:teuthology.orchestra.run.smithi071:osd.1> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.1.service 2021-11-20T07:36:01.893 INFO:tasks.cephadm:Deploying osd.2 on smithi071 with /dev/vg_nvme/lv_2... 2021-11-20T07:36:01.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T07:36:01.921 INFO:journalctl@ceph.osd.1.smithi071.stdout:-- Logs begin at Sat 2021-11-20 07:22:38 UTC. -- 2021-11-20T07:36:02.318 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[55662]: debug 2021-11-20T07:36:02.020+0000 7f03dadc2080 -1 Falling back to public interface 2021-11-20T07:36:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.562112+0000 2021-11-20T07:36:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: mon.smithi071 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.565846+0000 mon.smithi071 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.577446+0000 mon.smithi071 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.71:9093"}]: dispatch 2021-11-20T07:36:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.577826+0000 mgr.smithi071.ynqmwn (mgr.14162) 88 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.71:9093"}]: dispatch 2021-11-20T07:36:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.584975+0000 mon.smithi071 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.588875+0000 mon.smithi071 (mon.0) 254 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T07:36:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.589246+0000 mgr.smithi071.ynqmwn (mgr.14162) 89 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T07:36:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.596817+0000 mon.smithi071 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.71:3000"}]: dispatch 2021-11-20T07:36:02.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.597148+0000 mgr.smithi071.ynqmwn (mgr.14162) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.71:3000"}]: dispatch 2021-11-20T07:36:02.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.601167+0000 mon.smithi071 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:02.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.604879+0000 mon.smithi071 (mon.0) 257 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T07:36:02.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.605384+0000 mgr.smithi071.ynqmwn (mgr.14162) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T07:36:02.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.609516+0000 mon.smithi071 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.71:9095"}]: dispatch 2021-11-20T07:36:02.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.609831+0000 mgr.smithi071.ynqmwn (mgr.14162) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.71:9095"}]: dispatch 2021-11-20T07:36:02.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.614626+0000 mon.smithi071 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:02.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: audit 2021-11-20T07:36:01.618398+0000 mon.smithi071 (mon.0) 260 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:02.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: cluster 2021-11-20T07:36 2021-11-20T07:36:02.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:02 smithi071 conmon[33687]: :02.338011+0000 mgr.smithi071.ynqmwn (mgr.14162) 93 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:03.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:01.562112+0000 mon.smithi071 (mon. 2021-11-20T07:36:03.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 0) 250 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:03.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:01.565846+0000 mon.smithi071 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:03.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:36:03.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 11-20T07:36:01.577446+0000 mon.smithi071 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.71:9093"}]: dispatch 2021-11-20T07:36:03.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:03.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:01.577826+0000 mgr.smithi071.ynqmwn (mgr.14162) 88 : audit [DBG] 2021-11-20T07:36:03.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.71:9093"}]: dispatch 2021-11-20T07:36:03.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:03.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:01.584975+0000 mon.smithi071 (mon.0) 253 : audit [INF] 2021-11-20T07:36:03.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:03.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021 2021-11-20T07:36:03.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: -11-20T07:36:01.588875+0000 mon.smithi071 (mon.0) 254 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T07:36:03.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:03.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:36:03.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: :36:01.589246+0000 mgr.smithi071.ynqmwn (mgr.14162) 89 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T07:36:03.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:01 2021-11-20T07:36:03.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: .596817+0000 mon.smithi071 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.71:3000"}]: dispatch 2021-11-20T07:36:03.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:36:03.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 20T07:36:01.597148+0000 mgr.smithi071.ynqmwn (mgr.14162) 90 : 2021-11-20T07:36:03.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.71:3000"}]: dispatch 2021-11-20T07:36:03.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:01.601167+0000 mon.smithi071 ( 2021-11-20T07:36:03.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:03.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:03.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:01.604879+0000 mon.smithi071 (mon.0 2021-11-20T07:36:03.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: ) 257 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T07:36:03.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07: 2021-11-20T07:36:03.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 36:01.605384+0000 mgr.smithi071.ynqmwn (mgr.14162) 91 : audit [DBG] 2021-11-20T07:36:03.040 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T07:36:03.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021 2021-11-20T07:36:03.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: -11-20T07:36:01.609516+0000 mon.smithi071 (mon.0) 258 : audit [INF] 2021-11-20T07:36:03.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.71:9095"}]: dispatch 2021-11-20T07:36:03.041 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07:36:03.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:01.609831+0000 mgr.smithi071.ynqmwn (mgr.14162) 92 : audit [DBG] 2021-11-20T07:36:03.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.71:9095"}]: dispatch 2021-11-20T07:36:03.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:36:03.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 07:36:01.614626+0000 mon.smithi071 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:03.042 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: audit 2021-11-20T07 2021-11-20T07:36:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: :36:01.618398+0000 mon.smithi071 (mon.0) 260 : audit [DBG] 2021-11-20T07:36:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: cluster 2021-11-20T07:36:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: 2021-11-20T07:36:02.338011+0000 mgr.smithi071.ynqmwn (mgr.14162) 93 : cluster [DBG] 2021-11-20T07:36:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:02 smithi112 conmon[36062]: pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:04.068 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T07:36:04.069 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T07:36:04.069 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2021-11-20T07:36:04.069 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2021-11-20T07:36:04.071 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165148 s, 635 MB/s 2021-11-20T07:36:04.071 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 2021-11-20T07:36:04.071 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2021-11-20T07:36:04.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi071:vg_nvme/lv_2 2021-11-20T07:36:05.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:04 smithi112 conmon[36062]: audit 2021-11-20T07:36:03.763346+0000 mon.smithi071 (mon.0) 261 2021-11-20T07:36:05.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:04 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:05.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:04 smithi112 conmon[36062]: cluster 2021-11-20T07:36:04.338333+0000 mgr.smithi071.ynqmwn (mgr.14162) 94 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:05.060 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:04 smithi071 conmon[33687]: audit 2021-11-20T07:36:03.763346+0000 mon.smithi071 (mon.0) 261 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:05.061 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:04 smithi071 conmon[33687]: cluster 2021-11-20T07:36:04.338333+0000 mgr.smithi071.ynqmwn (mgr.14162) 94 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:05.657 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:36:05 smithi071 conmon[55662]: debug 2021-11-20T07:36:05.542+0000 7f03dadc2080 -1 osd.1 0 log_to_monitors {default=true} 2021-11-20T07:36:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- 172.21.15.71:0/1061338142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f77d0 msgr2=0x7f53240f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:06.037 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 --2- 172.21.15.71:0/1061338142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f77d0 0x7f53240f7c30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5320004660 tx=0x7f5320009b30).stop 2021-11-20T07:36:06.037 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- 172.21.15.71:0/1061338142 shutdown_connections 2021-11-20T07:36:06.038 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 --2- 172.21.15.71:0/1061338142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f77d0 0x7f53240f7c30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:06.038 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 --2- 172.21.15.71:0/1061338142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f6530 0x7f53240f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:06.038 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- 172.21.15.71:0/1061338142 >> 172.21.15.71:0/1061338142 conn(0x7f53240f1c90 msgr2=0x7f53240f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:06.039 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- 172.21.15.71:0/1061338142 shutdown_connections 2021-11-20T07:36:06.039 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- 172.21.15.71:0/1061338142 wait complete. 2021-11-20T07:36:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 Processor -- start 2021-11-20T07:36:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.030+0000 7f532a9af700 1 -- start start 2021-11-20T07:36:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f77d0 0x7f5324069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532406a380 con 0x7f53240f6530 2021-11-20T07:36:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532406a4c0 con 0x7f53240f77d0 2021-11-20T07:36:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47390/0 (socket says 172.21.15.71:47390) 2021-11-20T07:36:06.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 -- 172.21.15.71:0/890697295 learned_addr learned my addr 172.21.15.71:0/890697295 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:36:06.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53291ac700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f77d0 0x7f5324069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:06.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 -- 172.21.15.71:0/890697295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f77d0 msgr2=0x7f5324069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:06.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f77d0 0x7f5324069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:06.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 -- 172.21.15.71:0/890697295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320005040 con 0x7f53240f6530 2021-11-20T07:36:06.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53299ad700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f531800c530 tx=0x7f5318007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:06.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f531800f650 con 0x7f53240f6530 2021-11-20T07:36:06.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f531800fc60 con 0x7f53240f6530 2021-11-20T07:36:06.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5318012d20 con 0x7f53240f6530 2021-11-20T07:36:06.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532406a7a0 con 0x7f53240f6530 2021-11-20T07:36:06.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.031+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532410ed00 con 0x7f53240f6530 2021-11-20T07:36:06.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.032+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5324045ad0 con 0x7f53240f6530 2021-11-20T07:36:06.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.033+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5318019030 con 0x7f53240f6530 2021-11-20T07:36:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.033+0000 7f53167fc700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 0x7f531005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.033+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1997+0+0 (secure 0 0 0) 0x7f5318074e30 con 0x7f53240f6530 2021-11-20T07:36:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.033+0000 7f53291ac700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 0x7f531005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.033+0000 7f53291ac700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 0x7f531005e7c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5320013580 tx=0x7f5320009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:06.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.037+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5318048880 con 0x7f53240f6530 2021-11-20T07:36:06.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:06.221+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f53240fee40 con 0x7f531005c300 2021-11-20T07:36:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: audit 2021-11-20T07:36:05.546605+0000 mon.smithi071 (mon.0) 262 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T07:36:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:36:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: 20T07:36:06.224242+0000 mon.smithi071 (mon.0) 263 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:06.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: audit 2021-11-20T07:36:06.225625+0000 mon.smithi071 (mon.0) 264 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:06.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: audit 2021-11-20T07:36:06.226237+0000 2021-11-20T07:36:06.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:06 smithi112 conmon[36062]: mon.smithi071 (mon.0) 265 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:06.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:06 smithi071 conmon[33687]: audit 2021-11-20T07:36:05.546605+0000 mon.smithi071 (mon.0) 262 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T07:36:06.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:06 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.224242+0000 mon.smithi071 (mon.0) 263 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:06.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:06 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.225625+0000 mon.smithi071 (mon.0) 264 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:06.786 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:06 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.226237+0000 mon.smithi071 (mon.0) 265 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:07.431 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[55662]: debug 2021-11-20T07:36:07.341+0000 7f03d1333700 -1 osd.1 0 waiting for initial osdmap 2021-11-20T07:36:07.431 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[55662]: debug 2021-11-20T07:36:07.346+0000 7f03cdaaf700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:36:07.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.223505+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:36:07.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: 14162) 95 : audit [DBG] from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:07.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:06.225422+0000 mgr.smithi071.ynqmwn (mgr.14162) 96 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:36:07.432 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: cluster 2021-11-20T07:36:06.338591+0000 mgr.smithi071.ynqmwn (mgr.14162) 97 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:07.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.338734+0000 mon.smithi071 (mon.0) 266 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T07:36:07.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: cluster 2021-11-20T07:36:06.338809+0000 mon.smithi071 (mon.0) 267 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T07:36:07.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.338934+0000 mon.smithi071 (mon.0) 268 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:07.433 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: audit 2021-11-20T07:36:06.339353+0000 mon.smithi071 (mon 2021-11-20T07:36:07.434 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:07 smithi071 conmon[33687]: .0) 269 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:36:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: audit 2021-11-20T07:36:06.223505+0000 mgr.smithi071.ynqmwn (mgr.14162) 95 : audit [DBG] from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: cephadm 2021-11- 2021-11-20T07:36:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: 20T07:36:06.225422+0000 mgr.smithi071.ynqmwn (mgr.14162) 96 : cephadm [INF] Applying service osd.None on host smithi071... 2021-11-20T07:36:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: cluster 2021-11-20T07:36:06.338591+0000 mgr.smithi071.ynqmwn (mgr.14162) 97 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: audit 2021-11-20T07:36:06 2021-11-20T07:36:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: .338734+0000 mon.smithi071 (mon.0) 266 : audit 2021-11-20T07:36:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T07:36:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: cluster 2021-11-20T07:36: 2021-11-20T07:36:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: 06.338809+0000 mon.smithi071 (mon.0) 267 : cluster 2021-11-20T07:36:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T07:36:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: audit 2021-11-20T07:36 2021-11-20T07:36:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: :06.338934+0000 mon.smithi071 (mon.0) 268 : 2021-11-20T07:36:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: audit 2021-11-20T07: 2021-11-20T07:36:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: 36:06.339353+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:07.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:07 smithi112 conmon[36062]: 269 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:36:08.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: audit 2021-11-20T07:36:07.339732+0000 2021-11-20T07:36:08.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: mon.smithi071 (mon.0) 270 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:36:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: cluster 2021-11-20T07:36:07.339783+0000 mon.smithi071 (mon.0) 271 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T07:36:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: audit 2021-11-20T07:36:07.340113+0000 mon.smithi071 (mon.0) 272 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: audit 2021-11-20T07:36:07.343420+0000 mon.smithi071 (mon.0) 273 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:08 smithi071 conmon[33687]: audit 2021-11-20T07:36:07.396838+0000 mon.smithi071 (mon.0) 274 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: audit 2021-11-20T07:36:07.339732+0000 mon.smithi071 (mon.0) 270 : audit [INF] from='osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:36:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:36:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: :36:07.339783+0000 mon.smithi071 (mon.0) 271 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T07:36:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: audit 2021-11-20T07:36:07.340113+0000 mon.smithi071 (mon.0) 272 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: audit 2021-11-20T07:36:07.343420+0000 mon.smithi071 2021-11-20T07:36:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: (mon.0) 273 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:36:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: 20T07:36:07.396838+0000 mon.smithi071 (mon 2021-11-20T07:36:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:08 smithi112 conmon[36062]: .0) 274 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:09.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11-20T07:36:06.559818+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11-20T07:36:06. 2021-11-20T07:36:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 560026+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11-20T07:36:08.338839+0000 mgr.smithi071.ynqmwn (mgr.14162) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021-11-20T07:36: 2021-11-20T07:36:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 08.343073+0000 mon.smithi071 (mon.0) 275 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11-20T07:36:08. 2021-11-20T07:36:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 347232+0000 mon.smithi071 (mon.0) 276 : cluster 2021-11-20T07:36:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: [INF] osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] boot 2021-11-20T07:36:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:36:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 36:08.347263+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 277 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T07:36:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:36:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 07:36:08.347937+0000 mon.smithi071 (mon. 2021-11-20T07:36:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 0) 278 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:36:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 20T07:36:08.607158+0000 mon.smithi071 (mon 2021-11-20T07:36:09.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: .0) 279 : audit [INF] from='client.? 172.21.15.71:0/225743771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c011d0ec-7e59-44fb-bc53-eb3324d914bf"}]: dispatch 2021-11-20T07:36:09.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021-11-20T07:36:08. 2021-11-20T07:36:09.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 613815+0000 mon.smithi071 (mon.0) 280 : audit [INF] from='client.? 172.21.15.71:0/225743771' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c011d0ec-7e59-44fb-bc53-eb3324d914bf"}]': finished 2021-11-20T07:36:09.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:36:09.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: -20T07:36:08.613859+0000 mon.smithi071 (mon 2021-11-20T07:36:09.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: .0) 281 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T07:36:09.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:36:09.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 11-20T07:36:08.613959+0000 mon.smithi071 2021-11-20T07:36:09.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: (mon.0) 282 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:09.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:36:09.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: -20T07:36:09.020272+0000 mon.smithi071 (mon. 2021-11-20T07:36:09.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:09 smithi112 conmon[36062]: 0) 283 : audit [DBG] from='client.? 172.21.15.71:0/1007517594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:09.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11-20T07:36:06.559818+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:36:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: -20T07:36:06.560026+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11-20T07:36:08.338839+0000 mgr.smithi071.ynqmwn (mgr.14162) 98 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T07:36:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021-11-20T07:36:08.343073+0000 mon.smithi071 (mon.0) 275 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11-20T07:36:08.347232+0000 mon.smithi071 (mon.0) 276 : cluster [INF] osd.1 [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] boot 2021-11-20T07:36:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11-20T07:36:08.347263+0000 mon.smithi071 (mon.0) 277 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T07:36:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021-11-20T07:36:08.347937+0000 mon.smithi071 (mon.0) 278 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T07:36:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021-11-20T07:36:08.607158+0000 mon.smithi071 (mon.0) 279 : audit [INF] from='client.? 172.21.15.71:0/225743771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c011d0ec-7e59-44fb-bc53-eb3324d914bf"}]: dispatch 2021-11-20T07:36:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021-11-20T07:36:08.613815+0000 mon.smithi071 (mon.0) 280 : audit [INF] from='client.? 172.21.15.71:0/225743771' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c011d0ec-7e59-44fb-bc53-eb3324d914bf"}]': finished 2021-11-20T07:36:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: cluster 2021-11-20T07:36 2021-11-20T07:36:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: :08.613859+0000 mon.smithi071 (mon.0) 281 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T07:36:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021-11-20T07:36 2021-11-20T07:36:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: :08.613959+0000 mon.smithi071 (mon.0) 282 : audit 2021-11-20T07:36:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: audit 2021 2021-11-20T07:36:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: -11-20T07:36:09. 2021-11-20T07:36:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: 020272+0000 mon.smithi071 (mon.0) 283 2021-11-20T07:36:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:09 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1007517594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:11 smithi112 conmon[36062]: cluster 2021-11-20T07:36:10.339238+0000 mgr.smithi071.ynqmwn (mgr.14162) 99 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:11.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:11 smithi071 conmon[33687]: cluster 2021-11-20T07:36:10.339238+0000 mgr.smithi071.ynqmwn (mgr.14162) 99 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:12.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:12 smithi071 conmon[33687]: cluster 2021-11-20T07:36:12.339471+0000 mgr.smithi071.ynqmwn (mgr.14162) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:12 smithi112 conmon[36062]: cluster 2021-11-20T07:36:12.339471+0000 mgr.smithi071.ynqmwn (mgr.14162) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:14.452 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:14 smithi071 conmon[33687]: cluster 2021-11-20T07:36:14.339762+0000 mgr.smithi071.ynqmwn (mgr.14162) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:14 smithi112 conmon[36062]: cluster 2021-11-20T07:36:14.339762+0000 mgr.smithi071.ynqmwn (mgr.14162) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:16.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:14.582252+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:36:16.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: 102 : cephadm [INF] Detected new or changed devices on smithi071 2021-11-20T07:36:16.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: audit 2021-11-20T07:36:14.588799+0000 mon.smithi071 (mon.0) 284 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:16.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: audit 2021-11-20T07:36:14.592120+0000 mon.smithi071 (mon.0) 285 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:16.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: audit 2021-11-20T07:36:14.594284+0000 mon.smithi071 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:16.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: audit 2021-11-20T07:36:15.362780+0000 mon.smithi071 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T07:36:16.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: audit 2021-11-20T07:36:15.363358+0000 mon.smithi071 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:16.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:15 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:15.363782+0000 mgr.smithi071.ynqmwn (mgr.14162) 103 : cephadm [INF] Deploying daemon osd.2 on smithi071 2021-11-20T07:36:16.064 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:14.582252+0000 mgr.smithi071.ynqmwn (mgr.14162) 102 : 2021-11-20T07:36:16.064 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: cephadm [INF] Detected new or changed devices on smithi071 2021-11-20T07:36:16.064 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: audit 2021-11-20T07:36:14.588799+0000 mon.smithi071 (mon.0) 284 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:16.064 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: audit 2021-11-20T07:36:14.592120+0000 mon.smithi071 (mon.0) 285 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:16.065 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: audit 2021-11-20T07:36:14.594284+0000 mon.smithi071 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:16.065 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: audit 2021-11-20T07:36:15.362780+0000 mon.smithi071 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T07:36:16.065 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: audit 2021-11-20T07:36:15.363358+0000 mon.smithi071 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:16.065 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:15 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:15.363782+0000 mgr.smithi071.ynqmwn (mgr.14162) 103 : cephadm [INF] Deploying daemon osd.2 on smithi071 2021-11-20T07:36:16.845 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:16 smithi071 conmon[33687]: cluster 2021-11-20T07:36:16.339982+0000 mgr.smithi071.ynqmwn (mgr.14162) 104 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:17.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:16 smithi112 conmon[36062]: cluster 2021-11-20T07:36:16.339982+0000 mgr.smithi071.ynqmwn (mgr.14162) 104 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:17 smithi071 conmon[33687]: audit 2021-11-20T07:36:16.734094+0000 mon.smithi071 2021-11-20T07:36:17.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:17 smithi071 conmon[33687]: (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:18.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:17 smithi112 conmon[36062]: audit 2021-11-20T07:36:16.734094+0000 mon.smithi071 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:18.874 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:18 smithi071 conmon[33687]: cluster 2021-11-20T07:36:18.340257+0000 2021-11-20T07:36:18.875 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:18 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 105 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:19.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:18 smithi112 conmon[36062]: cluster 2021-11-20T07:36:18.340257+0000 mgr.smithi071.ynqmwn (mgr.14162) 105 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:19.047 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 2 on host 'smithi071' 2021-11-20T07:36:19.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.044+0000 7f53167fc700 1 -- 172.21.15.71:0/890697295 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f53240fee40 con 0x7f531005c300 2021-11-20T07:36:19.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 msgr2=0x7f531005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:19.050 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 0x7f531005e7c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5320013580 tx=0x7f5320009b30).stop 2021-11-20T07:36:19.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 msgr2=0x7f532406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:19.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f531800c530 tx=0x7f5318007e10).stop 2021-11-20T07:36:19.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 shutdown_connections 2021-11-20T07:36:19.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f531005c300 0x7f531005e7c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:19.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53240f6530 0x7f532406b9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:19.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 --2- 172.21.15.71:0/890697295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53240f77d0 0x7f5324069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 >> 172.21.15.71:0/890697295 conn(0x7f53240f1c90 msgr2=0x7f53240fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 shutdown_connections 2021-11-20T07:36:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:36:19.046+0000 7f532a9af700 1 -- 172.21.15.71:0/890697295 wait complete. 2021-11-20T07:36:19.499 DEBUG:teuthology.orchestra.run.smithi071:osd.2> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.2.service 2021-11-20T07:36:19.502 INFO:tasks.cephadm:Deploying osd.3 on smithi112 with /dev/vg_nvme/lv_4... 2021-11-20T07:36:19.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T07:36:19.530 INFO:journalctl@ceph.osd.2.smithi071.stdout:-- Logs begin at Sat 2021-11-20 07:22:38 UTC. -- 2021-11-20T07:36:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:19 smithi071 conmon[33687]: audit 2021-11-20T07:36:18.931021+0000 mon.smithi071 (mon.0) 290 2021-11-20T07:36:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:19 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:36:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:19 smithi071 conmon[33687]: audit 2021-11-20T07:36:18.931204+0000 mon.smithi071 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:36:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:19 smithi071 conmon[33687]: audit 2021-11-20T07:36:19.041062+0000 mon.smithi071 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:19 smithi071 conmon[33687]: audit 2021-11-20T07:36:19.045019+0000 mon.smithi071 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:19.949 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:19 smithi112 conmon[36062]: audit 2021-11-20T07:36:18.931021+0000 mon.smithi071 (mon.0 2021-11-20T07:36:19.949 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:19 smithi112 conmon[36062]: ) 290 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:36:19.950 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:19 smithi112 conmon[36062]: audit 2021-11-20T07:36:18.931204+0000 mon.smithi071 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:36:19.950 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:19 smithi112 conmon[36062]: audit 2021-11-20T07:36:19.041062+0000 mon.smithi071 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:19.950 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:19 smithi112 conmon[36062]: audit 2021-11-20T07:36:19.045019+0000 mon.smithi071 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:21.268 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T07:36:21.269 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T07:36:21.269 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2021-11-20T07:36:21.269 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2021-11-20T07:36:21.270 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110761 s, 947 MB/s 2021-11-20T07:36:21.270 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2021-11-20T07:36:21.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:21 smithi071 conmon[33687]: audit 2021-11-20T07:36:20.137887+0000 mon.smithi071 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:21.413 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:21 smithi071 conmon[33687]: cluster 2021-11-20T07:36:20.340806+0000 mgr.smithi071.ynqmwn (mgr.14162) 106 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:21.450 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:21 smithi112 conmon[36062]: audit 2021-11-20T07:36:20.137887+0000 mon.smithi071 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:21.450 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:21 smithi112 conmon[36062]: cluster 2021-11-20T07:36:20.340806+0000 mgr.smithi071.ynqmwn (mgr.14162) 106 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:21.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2021-11-20T07:36:22.472 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:22 smithi071 conmon[33687]: cluster 2021-11-20T07:36:22.341101+0000 mgr.smithi071.ynqmwn (mgr.14162) 107 : cluster [DBG] 2021-11-20T07:36:22.472 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:22 smithi071 conmon[33687]: pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:22.737 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:22 smithi112 conmon[36062]: cluster 2021-11-20T07:36:22.341101+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:36:22.737 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:22 smithi112 conmon[36062]: ) 107 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:23.153 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- 172.21.15.112:0/3318692850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f66c0 msgr2=0x7fc3180f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:23.154 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 --2- 172.21.15.112:0/3318692850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f66c0 0x7fc3180f6ae0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fc314004660 tx=0x7fc314009b30).stop 2021-11-20T07:36:23.154 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- 172.21.15.112:0/3318692850 shutdown_connections 2021-11-20T07:36:23.154 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 --2- 172.21.15.112:0/3318692850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f7960 0x7fc3180f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:36:23.155 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 --2- 172.21.15.112:0/3318692850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f66c0 0x7fc3180f6ae0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:23.155 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- 172.21.15.112:0/3318692850 >> 172.21.15.112:0/3318692850 conn(0x7fc3180f1e20 msgr2=0x7fc3180f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:23.155 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- 172.21.15.112:0/3318692850 shutdown_connections 2021-11-20T07:36:23.155 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- 172.21.15.112:0/3318692850 wait complete. 2021-11-20T07:36:23.156 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 Processor -- start 2021-11-20T07:36:23.156 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- start start 2021-11-20T07:36:23.156 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f66c0 0x7fc31806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:23.156 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31806a410 con 0x7fc3180f66c0 2021-11-20T07:36:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc320287700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31806a550 con 0x7fc3180f7960 2021-11-20T07:36:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc31d822700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc31d822700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:58092/0 (socket says 172.21.15.112:58092) 2021-11-20T07:36:23.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc31d822700 1 -- 172.21.15.112:0/2364203240 learned_addr learned my addr 172.21.15.112:0/2364203240 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:36:23.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.149+0000 7fc31e023700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f66c0 0x7fc31806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:23.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc31d822700 1 -- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f66c0 msgr2=0x7fc31806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:23.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc31d822700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f66c0 0x7fc31806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:23.158 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc31d822700 1 -- 172.21.15.112:0/2364203240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314005040 con 0x7fc3180f7960 2021-11-20T07:36:23.159 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc31d822700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc30800b410 tx=0x7fc308006ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:23.159 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3080125f0 con 0x7fc3180f7960 2021-11-20T07:36:23.159 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc30800be30 con 0x7fc3180f7960 2021-11-20T07:36:23.159 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31806a830 con 0x7fc3180f7960 2021-11-20T07:36:23.160 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc308002a20 con 0x7fc3180f7960 2021-11-20T07:36:23.160 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.150+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31810f010 con 0x7fc3180f7960 2021-11-20T07:36:23.160 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.151+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc318045ad0 con 0x7fc3180f7960 2021-11-20T07:36:23.160 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.151+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc30800f030 con 0x7fc3180f7960 2021-11-20T07:36:23.161 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.152+0000 7fc30effd700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 0x7fc30405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:23.161 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.152+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2429+0+0 (secure 0 0 0) 0x7fc308075720 con 0x7fc3180f7960 2021-11-20T07:36:23.161 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.152+0000 7fc31e023700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 0x7fc30405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:23.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.152+0000 7fc31e023700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 0x7fc30405e760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc314004e30 tx=0x7fc314009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:23.162 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.156+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc308048f70 con 0x7fc3180f7960 2021-11-20T07:36:23.318 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[62032]: debug 2021-11-20T07:36:23.021+0000 7f144c1dd080 -1 osd.2 0 log_to_monitors {default=true} 2021-11-20T07:36:23.339 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:23.337+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3180ff040 con 0x7fc30405c2a0 2021-11-20T07:36:23.761 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.025266+0000 mon.smithi071 (mon.0) 295 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T07:36:23.762 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.237404+0000 mon.smithi071 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:23.762 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.240619+0000 mon.smithi071 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:23.762 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.242826+0000 mon.smithi071 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:23.763 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.339292+0000 mgr.smithi071.ynqmwn (mgr.14162) 108 : audit [DBG] from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:23.763 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.340084+0000 mon.smithi071 (mon.0) 299 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:23.763 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:23.341184+0000 2021-11-20T07:36:23.763 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 109 : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:23.764 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.341405+0000 mon.smithi071 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:23.764 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:23 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.341927+0000 mon.smithi071 (mon.0) 301 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:23.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.025266+0000 mon.smithi071 (mon.0) 295 : audit 2021-11-20T07:36:23.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T07:36:23.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.237404+0000 mon.smithi071 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:23.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.240619+0000 mon.smithi071 (mon.0) 297 2021-11-20T07:36:23.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:23.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.242826+0000 mon.smithi071 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:23.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.339292+0000 mgr.smithi071.ynqmwn (mgr.14162) 108 : audit 2021-11-20T07:36:23.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: [DBG] from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:23.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:36:23.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: 07:36:23.340084+0000 mon.smithi071 (mon.0) 299 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:23.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:23.341184+0000 mgr.smithi071.ynqmwn 2021-11-20T07:36:23.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: (mgr.14162) 109 : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:23.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.341405+0000 mon.smithi071 ( 2021-11-20T07:36:23.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:23.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: 2021-11-20T07:36:23.341927+0000 mon.smithi071 (mon.0) 301 2021-11-20T07:36:23.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:23 smithi071 conmon[33687]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:24.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.431338+0000 mon.smithi071 (mon.0 2021-11-20T07:36:24.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: ) 302 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T07:36:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: cluster 2021-11-20T07:36:23.431384+0000 mon.smithi071 (mon.0) 303 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T07:36:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.431478+0000 mon.smithi071 (mon.0) 304 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: audit 2021-11-20T07:36:23.431773+0000 mon.smithi071 (mon.0) 305 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:36:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:24 smithi112 conmon[36062]: cluster 2021-11-20T07:36:24.341513+0000 mgr.smithi071.ynqmwn (mgr.14162) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:24.818 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[62032]: debug 2021-11-20T07:36:24.435+0000 7f144274e700 -1 osd.2 0 waiting for initial osdmap 2021-11-20T07:36:24.818 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[62032]: debug 2021-11-20T07:36:24.441+0000 7f143eeca700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:36:24.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.431338 2021-11-20T07:36:24.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 302 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T07:36:24.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: cluster 2021-11-20T07:36:23.431384+0000 mon.smithi071 (mon.0) 303 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T07:36:24.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.431478+0000 mon.smithi071 (mon.0) 304 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:24.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: audit 2021-11-20T07:36:23.431773+0000 mon.smithi071 (mon.0) 305 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2021-11-20T07:36:24.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:24 smithi071 conmon[33687]: cluster 2021-11-20T07:36:24.341513+0000 mgr.smithi071.ynqmwn (mgr.14162) 110 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:25.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:25 smithi112 conmon[36062]: audit 2021-11-20T07:36:24.432617+0000 mon.smithi071 (mon.0) 306 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:36:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:25 smithi112 conmon[36062]: cluster 2021-11-20T07:36:24.432684+0000 mon.smithi071 (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T07:36:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:25 smithi112 conmon[36062]: audit 2021-11-20T07:36:24.433041+0000 mon.smithi071 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:25 smithi112 conmon[36062]: audit 2021-11-20T07:36:24.438211+0000 mon.smithi071 (mon.0) 309 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:25 smithi071 conmon[33687]: audit 2021-11-20T07:36:24.432617+0000 mon.smithi071 (mon. 2021-11-20T07:36:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:25 smithi071 conmon[33687]: 0) 306 : audit [INF] from='osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2021-11-20T07:36:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:25 smithi071 conmon[33687]: cluster 2021-11-20T07:36:24.432684+0000 mon.smithi071 (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T07:36:25.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:25 smithi071 conmon[33687]: audit 2021-11-20T07:36:24.433041+0000 mon.smithi071 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:25.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:25 smithi071 conmon[33687]: audit 2021-11-20T07:36:24.438211+0000 mon.smithi071 (mon.0) 309 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:24.073346+0000 osd.2 (osd. 2021-11-20T07:36:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: 2) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:24.073471+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.436025+0000 mon.smithi071 (mon.0) 310 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:25.439038+0000 mon.smithi071 (mon.0) 311 : cluster [INF] osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] boot 2021-11-20T07:36:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:25.439069+0000 mon.smithi071 (mon.0) 312 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-20T07:36:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.439388+0000 mon.smithi071 (mon.0) 313 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:26.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.686606+0000 mon.smithi112 (mon.1) 2 : audit [INF] from='client.? 172.21.15.112:0/2617090784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]: dispatch 2021-11-20T07:36:26.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.687041+0000 mon.smithi071 (mon.0) 314 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]: dispatch 2021-11-20T07:36:26.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.690477+0000 mon.smithi071 (mon.0) 315 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]': finished 2021-11-20T07:36:26.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:25.690536+0000 mon.smithi071 (mon.0) 316 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-20T07:36:26.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.690656+0000 mon.smithi071 (mon.0) 317 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:26.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:25.919548+0000 mon.smithi071 (mon.0) 318 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:26.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: audit 2021-11-20T07:36:26.106586+0000 mon.smithi112 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.112:0/573211869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:26.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:26 smithi112 conmon[36062]: cluster 2021-11-20T07:36:26.341745+0000 mgr.smithi071.ynqmwn (mgr.14162) 111 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20T07:36:24.073346+0000 2021-11-20T07:36:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20T07:36:24.073471+0000 osd.2 (osd.2) 2021-11-20T07:36:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:36:26.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: -20T07:36:25.436025+0000 mon.smithi071 (mon.0) 310 : audit 2021-11-20T07:36:26.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:26.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20T07:36:25. 2021-11-20T07:36:26.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 439038+0000 mon.smithi071 (mon.0) 311 : cluster 2021-11-20T07:36:26.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [INF] osd.2 [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] boot 2021-11-20T07:36:26.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20T07:36:25. 2021-11-20T07:36:26.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 439069+0000 mon.smithi071 (mon.0) 312 : cluster 2021-11-20T07:36:26.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-20T07:36:26.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07:36:25 2021-11-20T07:36:26.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: .439388+0000 mon.smithi071 (mon.0) 313 : audit 2021-11-20T07:36:26.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T07:36:26.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07:36:25. 2021-11-20T07:36:26.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 686606+0000 mon.smithi112 (mon.1) 2 : audit 2021-11-20T07:36:26.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [INF] from='client.? 172.21.15.112:0/2617090784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]: dispatch 2021-11-20T07:36:26.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07:36:25. 2021-11-20T07:36:26.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 687041+0000 mon.smithi071 (mon.0) 314 : audit 2021-11-20T07:36:26.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]: dispatch 2021-11-20T07:36:26.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07:36:25.690477 2021-11-20T07:36:26.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 315 : audit 2021-11-20T07:36:26.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5fe79473-cc34-4424-876a-09b500aae5a4"}]': finished 2021-11-20T07:36:26.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20T07:36:25.690536+0000 2021-11-20T07:36:26.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: mon.smithi071 (mon.0) 316 : cluster [DBG] 2021-11-20T07:36:26.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: osdmap e19: 4 total, 3 up, 4 in 2021-11-20T07:36:26.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07:36:26.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 2021-11-20T07:36:25.690656+0000 mon.smithi071 (mon.0 2021-11-20T07:36:26.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: ) 317 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:26.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:36:26.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: :36:25.919548+0000 mon.smithi071 (mon.0 2021-11-20T07:36:26.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: ) 318 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:26.827 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:36:26.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 36:26.106586+0000 mon.smithi112 (mon.1) 2021-11-20T07:36:26.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: 3 : audit [DBG] from='client.? 172.21.15.112:0/573211869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:26.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:36:26.828 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: T07:36:26.341745+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:36:26.829 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:26 smithi071 conmon[33687]: .14162) 111 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T07:36:28.277 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:27 smithi071 conmon[33687]: cluster 2021-11-20T07:36:26.927496+0000 2021-11-20T07:36:28.278 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:27 smithi071 conmon[33687]: mon.smithi071 (mon.0) 319 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T07:36:28.278 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:27 smithi071 conmon[33687]: audit 2021-11-20T07:36:26.927599+0000 mon.smithi071 (mon.0) 320 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:28.278 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:27 smithi071 conmon[33687]: audit 2021-11-20T07:36:27.839328+0000 mon.smithi071 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:28.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:27 smithi112 conmon[36062]: cluster 2021-11-20T07:36:26.927496+0000 mon.smithi071 (mon.0) 319 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T07:36:28.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:27 smithi112 conmon[36062]: audit 2021-11-20T07:36:26.927599+0000 mon.smithi071 (mon.0) 320 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:28.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:27 smithi112 conmon[36062]: audit 2021-11-20T07:36:28.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:27 smithi112 conmon[36062]: 2021-11-20T07:36:27.839328+0000 mon.smithi071 (mon.0 2021-11-20T07:36:28.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:27 smithi112 conmon[36062]: ) 321 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:29.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:28 smithi071 conmon[33687]: cluster 2021-11-20T07:36:28.342034+0000 2021-11-20T07:36:29.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:28 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 112 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:29.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:28 smithi112 conmon[36062]: cluster 2021-11-20T07:36:28.342034+0000 mgr.smithi071.ynqmwn (mgr.14162) 112 : cluster 2021-11-20T07:36:29.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:28 smithi112 conmon[36062]: [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:31.674 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:31 smithi112 conmon[36062]: cluster 2021-11-20T07:36:30.342531+0000 mgr.smithi071.ynqmwn (mgr.14162) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:31.674 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:31 smithi112 conmon[36062]: audit 2021-11-20T07:36:30.831322+0000 mon.smithi071 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:31.674 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:31 smithi112 conmon[36062]: audit 2021-11-20T07:36:31.176072+0000 mon.smithi071 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:31 smithi071 conmon[33687]: cluster 2021-11-20T07:36:30.342531 2021-11-20T07:36:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:31 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:31 smithi071 conmon[33687]: audit 2021-11-20T07:36:30.831322+0000 mon.smithi071 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:31.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:31 smithi071 conmon[33687]: audit 2021-11-20T07:36:31.176072+0000 mon.smithi071 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:32.656 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:32 smithi112 conmon[36062]: audit 2021-11-20T07:36:31.875632+0000 mon.smithi071 ( 2021-11-20T07:36:32.656 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:32 smithi112 conmon[36062]: mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T07:36:32.657 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:32 smithi112 conmon[36062]: audit 2021-11-20T07:36:31.876429+0000 mon.smithi071 (mon.0) 325 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:32 smithi071 conmon[33687]: audit 2021-11-20T07:36:31.875632+0000 mon.smithi071 (mon.0) 324 2021-11-20T07:36:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:32 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T07:36:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:32 smithi071 conmon[33687]: audit 2021-11-20T07:36:31.876429+0000 mon.smithi071 (mon.0) 325 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:33.604 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:33 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:31.877089+0000 mgr.smithi071.ynqmwn (mgr.14162) 114 : cephadm [INF] Deploying daemon osd.3 on smithi112 2021-11-20T07:36:33.605 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:33 smithi112 conmon[36062]: cluster 2021-11-20T07:36:32.342871+0000 mgr.smithi071.ynqmwn (mgr.14162) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:33 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:31.877089+0000 mgr.smithi071.ynqmwn (mgr.14162) 114 : 2021-11-20T07:36:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:33 smithi071 conmon[33687]: cephadm [INF] Deploying daemon osd.3 on smithi112 2021-11-20T07:36:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:33 smithi071 conmon[33687]: cluster 2021-11-20T07:36:32.342871+0000 mgr.smithi071.ynqmwn (mgr.14162) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:35.061 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 3 on host 'smithi112' 2021-11-20T07:36:35.062 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.059+0000 7fc30effd700 1 -- 172.21.15.112:0/2364203240 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc3180ff040 con 0x7fc30405c2a0 2021-11-20T07:36:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 msgr2=0x7fc30405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 0x7fc30405e760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc314004e30 tx=0x7fc314009b30).stop 2021-11-20T07:36:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 msgr2=0x7fc318069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc30800b410 tx=0x7fc308006ef0).stop 2021-11-20T07:36:35.063 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 shutdown_connections 2021-11-20T07:36:35.064 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc30405c2a0 0x7fc30405e760 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:35.064 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3180f66c0 0x7fc31806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:35.064 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 --2- 172.21.15.112:0/2364203240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3180f7960 0x7fc318069e20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:35.064 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 >> 172.21.15.112:0/2364203240 conn(0x7fc3180f1e20 msgr2=0x7fc3180fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:35.065 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 shutdown_connections 2021-11-20T07:36:35.065 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:35.061+0000 7fc320287700 1 -- 172.21.15.112:0/2364203240 wait complete. 2021-11-20T07:36:35.449 DEBUG:teuthology.orchestra.run.smithi112:osd.3> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.3.service 2021-11-20T07:36:35.451 INFO:tasks.cephadm:Deploying osd.4 on smithi112 with /dev/vg_nvme/lv_3... 2021-11-20T07:36:35.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T07:36:35.481 INFO:journalctl@ceph.osd.3.smithi112.stdout:-- Logs begin at Sat 2021-11-20 07:22:39 UTC. -- 2021-11-20T07:36:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: cluster 2021-11-20T07:36:34.343181+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:36:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: 14162) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: audit 2021-11-20T07:36:35.047896+0000 mon.smithi071 (mon.0) 326 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: audit 2021-11-20T07:36:35.051552+0000 mon.smithi071 (mon.0) 327 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: audit 2021-11-20T07:36:35.058614+0000 mon.smithi071 (mon.0) 328 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:35 smithi112 conmon[36062]: audit 2021-11-20T07:36:35.062503+0000 mon.smithi071 (mon.0) 329 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T07:36:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: cluster 2021-11-20T07:36:34.343181+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:36:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: 14162) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: audit 2021-11-20T07:36:35.047896+0000 mon.smithi071 (mon.0) 326 : audit [DBG] 2021-11-20T07:36:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:36:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: :36:35.051552+0000 mon.smithi071 ( 2021-11-20T07:36:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: mon.0) 327 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: 2021-11-20T07:36:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: audit 2021 2021-11-20T07:36:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: -11-20T07:36:35 2021-11-20T07:36:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: .058614+0000 mon.smithi071 (mon.0 2021-11-20T07:36:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: ) 328 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: audit 2021 2021-11-20T07:36:35.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: -11-20T07:36:35 2021-11-20T07:36:35.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: .062503+0000 mon.smithi071 (mon. 2021-11-20T07:36:35.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: 0) 329 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T07:36:35.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:35 smithi071 conmon[33687]: 2021-11-20T07:36:37.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: audit 2021-11-20T07:36:36.059431+0000 mon.smithi071 (mon. 2021-11-20T07:36:37.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: 0) 330 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T07:36:37.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: cluster 2021-11-20T07:36:36.059482+0000 mon.smithi071 (mon.0) 331 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T07:36:37.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: audit 2021-11-20T07:36:36.059584+0000 mon.smithi071 (mon.0) 332 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:37.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: audit 2021-11-20T07:36:36.060153+0000 mon.smithi071 (mon.0) 333 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T07:36:37.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:37 smithi112 conmon[36062]: cluster 2021-11-20T07:36:36.343518+0000 mgr.smithi071.ynqmwn (mgr.14162) 117 : cluster [DBG] pgmap v80: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:37.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:37 smithi071 conmon[33687]: audit 2021-11-20T07:36:36.059431+0000 mon.smithi071 (mon.0) 330 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T07:36:37.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:37 smithi071 conmon[33687]: cluster 2021-11-20T07:36:36.059482+0000 mon.smithi071 (mon.0) 331 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T07:36:37.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:37 smithi071 conmon[33687]: audit 2021-11-20T07:36:36.059584+0000 mon.smithi071 (mon.0) 332 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:37.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:37 smithi071 conmon[33687]: audit 2021-11-20T07:36:36.060153+0000 mon.smithi071 (mon.0) 333 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T07:36:37.321 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:37 smithi071 conmon[33687]: cluster 2021-11-20T07:36:36.343518+0000 mgr.smithi071.ynqmwn (mgr.14162) 117 : cluster [DBG] pgmap v80: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:37.887 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T07:36:37.889 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T07:36:37.889 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2021-11-20T07:36:37.889 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2021-11-20T07:36:37.890 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137787 s, 761 MB/s 2021-11-20T07:36:37.890 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2021-11-20T07:36:38.140 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:38 smithi112 conmon[36062]: audit 2021-11-20T07:36:37.064115+0000 mon.smithi071 (mon.0) 334 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T07:36:38.140 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:38 smithi112 conmon[36062]: cluster 2021-11-20T07:36:37.064182+0000 mon.smithi071 (mon.0) 335 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T07:36:38.140 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:38 smithi112 conmon[36062]: audit 2021-11-20T07:36:37.064427+0000 mon.smithi071 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:38.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: audit 2021-11-20T07:36:37.064115+0000 mon.smithi071 (mon.0 2021-11-20T07:36:38.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: ) 334 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T07:36:38.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: cluster 2021-11-20T07:36:37.064182+0000 mon.smithi071 (mon.0) 335 : cluster [DBG] 2021-11-20T07:36:38.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: osdmap e22: 4 total, 3 up, 4 in 2021-11-20T07:36:38.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: audit 2021-11-20T07:36:38.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: 2021-11-20T07:36:37.064427+0000 2021-11-20T07:36:38.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: mon.smithi071 (mon.0) 336 : audit 2021-11-20T07:36:38.320 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:38 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:38.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2021-11-20T07:36:39.074 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:36:39 smithi112 conmon[42410]: debug 2021-11-20T07:36:39.008+0000 7fc838d0a080 -1 osd.3 0 log_to_monitors {default=true} 2021-11-20T07:36:39.412 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:39 smithi112 conmon[36062]: cluster 2021-11-20T07:36:38.343796+0000 mgr.smithi071.ynqmwn (mgr.14162) 118 : cluster [DBG] pgmap v82: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:39.412 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:39 smithi112 conmon[36062]: audit 2021-11-20T07:36:38.729933+0000 mon.smithi071 (mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:39.413 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:39 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.011254+0000 mon.smithi112 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T07:36:39.413 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:39 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.011642+0000 mon.smithi071 (mon.0) 338 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T07:36:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:39 smithi071 conmon[33687]: cluster 2021-11-20T07:36:38.343796+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:36:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:39 smithi071 conmon[33687]: ) 118 : cluster [DBG] pgmap v82: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:39 smithi071 conmon[33687]: audit 2021-11-20T07:36:38.729933+0000 mon.smithi071 (mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:39 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.011254+0000 mon.smithi112 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T07:36:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:39 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.011642+0000 mon.smithi071 (mon.0) 338 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T07:36:40.395 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[42410]: debug 2021-11-20T07:36:40.082+0000 7fc82f27b700 -1 osd.3 0 waiting for initial osdmap 2021-11-20T07:36:40.395 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[42410]: debug 2021-11-20T07:36:40.088+0000 7fc8299f3700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:36:40.396 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.076068+0000 mon.smithi071 (mon.0) 339 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T07:36:40.396 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: cluster 2021-11-20T07:36:39.076125+0000 mon.smithi071 (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T07:36:40.396 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.076232+0000 mon.smithi071 (mon.0) 341 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:40.396 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.076672+0000 mon.smithi112 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:40.396 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.076977+0000 mon.smithi071 (mon.0) 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:40.397 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:40 smithi112 conmon[36062]: audit 2021-11-20T07:36:39.562898+0000 mon.smithi071 (mon.0) 343 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:40.432 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- 172.21.15.112:0/3067542201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 msgr2=0x7ff2900f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:40.432 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 --2- 172.21.15.112:0/3067542201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff2900f6950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff284004660 tx=0x7ff284009b30).stop 2021-11-20T07:36:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- 172.21.15.112:0/3067542201 shutdown_connections 2021-11-20T07:36:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 --2- 172.21.15.112:0/3067542201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 0x7ff2900f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 --2- 172.21.15.112:0/3067542201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff2900f6950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- 172.21.15.112:0/3067542201 >> 172.21.15.112:0/3067542201 conn(0x7ff2900f1c90 msgr2=0x7ff2900f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- 172.21.15.112:0/3067542201 shutdown_connections 2021-11-20T07:36:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- 172.21.15.112:0/3067542201 wait complete. 2021-11-20T07:36:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 Processor -- start 2021-11-20T07:36:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.429+0000 7ff295efb700 1 -- start start 2021-11-20T07:36:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 0x7ff290107300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff290107900 con 0x7ff2900f77d0 2021-11-20T07:36:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff290108110 con 0x7ff2900f6530 2021-11-20T07:36:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:40.436 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:58144/0 (socket says 172.21.15.112:58144) 2021-11-20T07:36:40.436 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 -- 172.21.15.112:0/3435618317 learned_addr learned my addr 172.21.15.112:0/3435618317 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:36:40.436 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff28ffff700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 0x7ff290107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 -- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 msgr2=0x7ff290107300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 0x7ff290107300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 -- 172.21.15.112:0/3435618317 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284005040 con 0x7ff2900f6530 2021-11-20T07:36:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff294ef9700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff284004e30 tx=0x7ff284009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff284018070 con 0x7ff2900f6530 2021-11-20T07:36:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff290108330 con 0x7ff2900f6530 2021-11-20T07:36:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff284013410 con 0x7ff2900f6530 2021-11-20T07:36:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28401c3c0 con 0x7ff2900f6530 2021-11-20T07:36:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.430+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2901087b0 con 0x7ff2900f6530 2021-11-20T07:36:40.440 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.431+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff290045ad0 con 0x7ff2900f6530 2021-11-20T07:36:40.440 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.432+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff28401d420 con 0x7ff2900f6530 2021-11-20T07:36:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.432+0000 7ff28d7fa700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 0x7ff28005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.432+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3721+0+0 (secure 0 0 0) 0x7ff284077490 con 0x7ff2900f6530 2021-11-20T07:36:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.432+0000 7ff28ffff700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 0x7ff28005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.433+0000 7ff28ffff700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 0x7ff28005e7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff27c0096c0 tx=0x7ff27c009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.436+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff28404a7d0 con 0x7ff2900f6530 2021-11-20T07:36:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.076068+0000 mon.smithi071 (mon.0 2021-11-20T07:36:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: ) 339 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T07:36:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: cluster 2021-11-20T07:36:39.076125+0000 mon.smithi071 (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T07:36:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.076232+0000 mon.smithi071 (mon 2021-11-20T07:36:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: .0) 341 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.076672+0000 mon.smithi112 (mon.1) 5 : audit [INF] 2021-11-20T07:36:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: from='osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:40.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: audit 2021-11-20T07:36:39.076977 2021-11-20T07:36:40.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:40.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:40 smithi071 conmon[33687]: 2021-11-20T07:36:39.562898+0000 mon.smithi071 (mon.0) 343 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:40.625 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:40.623+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2901089d0 con 0x7ff28005c300 2021-11-20T07:36:41.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.078145+0000 mon.smithi071 (mon 2021-11-20T07:36:41.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: .0) 344 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:36:41.348 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.078271+0000 mon.smithi071 (mon.0) 345 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T07:36:41.348 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.078655+0000 mon.smithi071 (mon.0) 346 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.348 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.083734+0000 mon.smithi071 (mon.0) 347 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.349 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.344110+0000 mgr.smithi071.ynqmwn (mgr.14162) 119 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:41.349 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.345072+0000 mon.smithi071 (mon.0) 348 : cluster [INF] osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] boot 2021-11-20T07:36:41.350 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.345122+0000 mon.smithi071 (mon.0) 349 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-11-20T07:36:41.350 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.345864+0000 mon.smithi071 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.351 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.626942+0000 mon.smithi071 (mon.0) 351 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:41.351 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.628596+0000 mon.smithi071 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:41.351 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:41 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.629312+0000 mon.smithi071 (mon.0) 353 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.078145 2021-11-20T07:36:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 344 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:36:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: cluster 2021-11-20T07:36:40.078271+0000 mon.smithi071 (mon.0) 345 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T07:36:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.078655+0000 mon.smithi071 (mon.0) 346 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021 2021-11-20T07:36:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: -11-20T07:36:40.083734+0000 mon.smithi071 (mon.0) 347 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: cluster 2021-11-20T07:36:40.344110+0000 mgr.smithi071.ynqmwn 2021-11-20T07:36:41.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: (mgr.14162) 119 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T07:36:41.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: cluster 2021-11-20T07:36:40.345072+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:41.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: 348 : cluster [INF] osd.3 [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] boot 2021-11-20T07:36:41.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:36:41.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: 11-20T07:36:40.345122+0000 mon.smithi071 (mon.0) 349 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-11-20T07:36:41.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.345864 2021-11-20T07:36:41.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T07:36:41.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.626942+0000 mon.smithi071 (mon.0) 351 : audit [DBG] 2021-11-20T07:36:41.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:41.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:36:41.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: -20T07:36:40.628596+0000 mon.smithi071 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:41.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.629312+0000 mon.smithi071 (mon. 2021-11-20T07:36:41.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:41 smithi071 conmon[33687]: 0) 353 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:42 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.003643+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:42 smithi112 conmon[36062]: cluster 2021-11-20T07:36:40.003787+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:42 smithi112 conmon[36062]: audit 2021-11-20T07:36:40.625979+0000 mgr.smithi071.ynqmwn (mgr.14162) 120 : audit [DBG] from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:42 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:40.628314+0000 mgr.smithi071.ynqmwn (mgr.14162) 121 : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:42 smithi112 conmon[36062]: cluster 2021-11-20T07:36:41.348351+0000 mon.smithi071 (mon.0) 354 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-20T07:36:42.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: cluster 2021-11-20T07:36:40.003643+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:42.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: 2021-11-20T07:36:42.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: cluster 2021-11-20T07:36:40.003787+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: audit 2021-11-20T07:36:40.625979+0000 mgr.smithi071.ynqmwn (mgr.14162) 120 : audit [DBG] from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: 2021-11-20T07:36:40.628314+0000 mgr.smithi071.ynqmwn (mgr.14162) 121 : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: cluster 2021-11-20T07:36:42.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: 2021-11-20T07:36:41.348351+0000 mon.smithi071 (mon.0) 354 : cluster [DBG] 2021-11-20T07:36:42.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:42 smithi071 conmon[33687]: osdmap e26: 4 total, 4 up, 4 in 2021-11-20T07:36:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: cluster 2021-11-20T07:36: 2021-11-20T07:36:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 42.344554+0000 mgr.smithi071.ynqmwn (mgr.14162) 122 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: cluster 2021-11-20T07:36:42.353365+0000 mon.smithi071 (mon.0) 355 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T07:36:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021-11-20T07:36:42.431891+0000 mon.smithi071 (mon.0) 356 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:43.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 2021-11-20T07:36:43.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:36:43.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 20T07:36:42.435526+0000 mon.smithi071 2021-11-20T07:36:43.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: (mon.0) 357 : audit 2021-11-20T07:36:43.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:43.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021 2021-11-20T07:36:43.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: -11-20T07:36:42 2021-11-20T07:36:43.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: .438627+0000 mon.smithi071 (mon. 2021-11-20T07:36:43.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 0) 358 : audit [DBG] 2021-11-20T07:36:43.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:43.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021-11-20T07:36:43.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 2021-11-20T07: 2021-11-20T07:36:43.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 36:43.132370+0000 mon.smithi112 ( 2021-11-20T07:36:43.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: mon.1) 6 : audit 2021-11-20T07:36:43.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: [INF] from='client.? 172.21.15.112:0/20452079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]: dispatch 2021-11-20T07:36:43.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:36:43.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: -20T07:36:43.132798 2021-11-20T07:36:43.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 2021-11-20T07:36:43.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]: dispatch 2021-11-20T07:36:43.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021 2021-11-20T07:36:43.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: -11-20T07:36: 2021-11-20T07:36:43.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 43.136743+0000 mon.smithi071 (mon 2021-11-20T07:36:43.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: .0) 360 : audit [INF] 2021-11-20T07:36:43.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]': finished 2021-11-20T07:36:43.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: cluster 2021-11-20T07:36:43.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 2021-11-20T07: 2021-11-20T07:36:43.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: 36:43.136819+0000 mon.smithi071 2021-11-20T07:36:43.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: (mon.0) 361 : cluster 2021-11-20T07:36:43.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T07:36:43.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: audit 2021 2021-11-20T07:36:43.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: -11-20T07:36:43 2021-11-20T07:36:43.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: .136918+0000 mon.smithi071 (mon 2021-11-20T07:36:43.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: .0) 362 : audit [DBG] 2021-11-20T07:36:43.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:43 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:43.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: cluster 2021-11-20T07:36:42.344554+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:36:43.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: ) 122 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:43.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: cluster 2021-11-20T07:36:42.353365+0000 mon.smithi071 (mon.0) 355 : 2021-11-20T07:36:43.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T07:36:43.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36:42 2021-11-20T07:36:43.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: .431891+0000 mon.smithi071 (mon.0) 356 : audit 2021-11-20T07:36:43.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:43.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36:42 2021-11-20T07:36:43.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: .435526+0000 mon.smithi071 (mon.0) 357 : audit 2021-11-20T07:36:43.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:43.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36: 2021-11-20T07:36:43.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: 42.438627+0000 mon.smithi071 (mon.0) 358 : audit 2021-11-20T07:36:43.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:43.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36: 2021-11-20T07:36:43.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: 43.132370+0000 mon.smithi112 (mon.1) 6 : 2021-11-20T07:36:43.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit [INF] from='client.? 172.21.15.112:0/20452079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]: dispatch 2021-11-20T07:36:43.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36: 2021-11-20T07:36:43.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: 43.132798+0000 mon.smithi071 (mon.0) 359 : 2021-11-20T07:36:43.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]: dispatch 2021-11-20T07:36:43.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11-20T07:36: 2021-11-20T07:36:43.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: 43.136743+0000 mon.smithi071 (mon.0) 360 : 2021-11-20T07:36:43.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "438e6bcc-10d6-473a-8732-7e2f399906d7"}]': finished 2021-11-20T07:36:43.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: cluster 2021-11-20T07:36:43.136819+0000 2021-11-20T07:36:43.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: mon.smithi071 (mon.0) 361 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T07:36:43.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:36:43.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: 20T07:36:43.136918+0000 mon.smithi071 ( 2021-11-20T07:36:43.826 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:43 smithi071 conmon[33687]: mon.0) 362 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:44.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:44 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:42.427243+0000 mgr.smithi071.ynqmwn (mgr.14162) 123 : cephadm [INF] 2021-11-20T07:36:44.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:44 smithi112 conmon[36062]: Detected new or changed devices on smithi112 2021-11-20T07:36:44.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:44 smithi112 conmon[36062]: audit 2021-11-20T07:36:43.551524+0000 mon.smithi112 (mon.1) 2021-11-20T07:36:44.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:44 smithi112 conmon[36062]: 7 : audit [DBG] from='client.? 172.21.15.112:0/1764788403' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:44.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:44 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:42.427243+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:36:44.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:44 smithi071 conmon[33687]: 14162) 123 : cephadm [INF] Detected new or changed devices on smithi112 2021-11-20T07:36:44.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:44 smithi071 conmon[33687]: audit 2021-11-20T07:36:43.551524+0000 mon.smithi112 (mon.1) 7 2021-11-20T07:36:44.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:44 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.112:0/1764788403' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:36:45.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:45 smithi112 conmon[36062]: cluster 2021-11-20T07:36:44.344869+0000 mgr.smithi071.ynqmwn (mgr.14162) 124 : cluster [DBG] 2021-11-20T07:36:45.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:45 smithi112 conmon[36062]: pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:45.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:45 smithi071 conmon[33687]: cluster 2021-11-20T07:36:44.344869+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:36:45.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:45 smithi071 conmon[33687]: .14162) 124 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:47.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:47 smithi112 conmon[36062]: cluster 2021-11-20T07:36:46.345252+0000 mgr.smithi071.ynqmwn (mgr.14162) 125 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:47.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:47 smithi112 conmon[36062]: audit 2021-11-20T07:36:47.241042+0000 mon.smithi071 (mon.0) 363 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:47.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:47 smithi112 conmon[36062]: audit 2021-11-20T07:36:47.244043+0000 mon.smithi071 (mon.0) 364 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:47 smithi071 conmon[33687]: cluster 2021-11-20T07:36:46.345252+0000 mgr.smithi071.ynqmwn (mgr.14162) 125 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:47 smithi071 conmon[33687]: audit 2021-11-20T07:36:47.241042+0000 mon.smithi071 (mon.0) 363 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:47 smithi071 conmon[33687]: audit 2021-11-20T07:36:47.244043+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:47.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:47 smithi071 conmon[33687]: 364 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:49.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:49 smithi112 conmon[36062]: cluster 2021-11-20T07:36:48.345554+0000 mgr.smithi071.ynqmwn (mgr.14162) 126 2021-11-20T07:36:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:49 smithi112 conmon[36062]: : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:49 smithi112 conmon[36062]: audit 2021-11-20T07:36:49.291067+0000 mon.smithi071 (mon.0) 365 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T07:36:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:49 smithi112 conmon[36062]: audit 2021-11-20T07:36:49.291958+0000 mon.smithi071 (mon.0) 366 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:49 smithi071 conmon[33687]: cluster 2021-11-20T07:36:48.345554+0000 mgr.smithi071.ynqmwn (mgr.14162) 126 : cluster [DBG] 2021-11-20T07:36:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:49 smithi071 conmon[33687]: pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:49 smithi071 conmon[33687]: audit 2021-11-20T07:36:49.291067+0000 mon.smithi071 (mon.0) 365 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T07:36:49.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:49 smithi071 conmon[33687]: audit 2021-11-20T07:36:49.291958+0000 mon.smithi071 (mon.0) 366 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:50.697 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:50 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:49.292667+0000 mgr.smithi071.ynqmwn (mgr.14162) 127 : cephadm [INF] Deploying daemon osd.4 on smithi112 2021-11-20T07:36:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:50 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:49.292667+0000 mgr.smithi071.ynqmwn (mgr.14162) 127 : cephadm 2021-11-20T07:36:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:50 smithi071 conmon[33687]: [INF] Deploying daemon osd.4 on smithi112 2021-11-20T07:36:51.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:51 smithi112 conmon[36062]: cluster 2021-11-20T07:36:50.345924+0000 mgr.smithi071.ynqmwn (mgr.14162) 128 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:51 smithi071 conmon[33687]: cluster 2021-11-20T07:36:50.345924+0000 mgr.smithi071.ynqmwn (mgr.14162) 128 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:51.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:51 smithi071 conmon[33687]: 2021-11-20T07:36:52.273 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 4 on host 'smithi112' 2021-11-20T07:36:52.275 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.270+0000 7ff28d7fa700 1 -- 172.21.15.112:0/3435618317 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff2901089d0 con 0x7ff28005c300 2021-11-20T07:36:52.275 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.272+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 msgr2=0x7ff28005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:52.276 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.272+0000 7ff295efb700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 0x7ff28005e7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff27c0096c0 tx=0x7ff27c009be0).stop 2021-11-20T07:36:52.276 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.272+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 msgr2=0x7ff290106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:52.276 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.272+0000 7ff295efb700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff284004e30 tx=0x7ff284009b30).stop 2021-11-20T07:36:52.276 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 shutdown_connections 2021-11-20T07:36:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2900f6530 0x7ff290106e10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff28005c300 0x7ff28005e7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 --2- 172.21.15.112:0/3435618317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2900f77d0 0x7ff290107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 >> 172.21.15.112:0/3435618317 conn(0x7ff2900f1c90 msgr2=0x7ff2900fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:52.277 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 shutdown_connections 2021-11-20T07:36:52.278 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:52.273+0000 7ff295efb700 1 -- 172.21.15.112:0/3435618317 wait complete. 2021-11-20T07:36:52.522 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:52 smithi112 conmon[36062]: audit 2021-11-20T07:36:52.261001+0000 mon.smithi071 (mon.0) 367 : audit 2021-11-20T07:36:52.522 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:52 smithi112 conmon[36062]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:52.523 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:52 smithi112 conmon[36062]: audit 2021-11-20T07:36:52.265617+0000 mon.smithi071 (mon.0) 368 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:52.523 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:52 smithi112 conmon[36062]: audit 2021-11-20T07:36:52.270934+0000 mon.smithi071 (mon.0) 369 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:52.773 DEBUG:teuthology.orchestra.run.smithi112:osd.4> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.4.service 2021-11-20T07:36:52.775 INFO:tasks.cephadm:Deploying osd.5 on smithi112 with /dev/vg_nvme/lv_2... 2021-11-20T07:36:52.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T07:36:52.813 INFO:journalctl@ceph.osd.4.smithi112.stdout:-- Logs begin at Sat 2021-11-20 07:22:39 UTC. -- 2021-11-20T07:36:52.817 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:52 smithi071 conmon[33687]: audit 2021-11-20T07:36:52.261001+0000 mon.smithi071 (mon 2021-11-20T07:36:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:52 smithi071 conmon[33687]: .0) 367 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:52 smithi071 conmon[33687]: audit 2021-11-20T07:36:52.265617+0000 mon.smithi071 (mon.0) 368 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:52 smithi071 conmon[33687]: audit 2021-11-20T07:36:52.270934+0000 mon.smithi071 (mon.0) 369 2021-11-20T07:36:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:52 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:53.740 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:53 smithi112 conmon[36062]: cluster 2021-11-20T07:36:52.346278+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:36:53.740 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:53 smithi112 conmon[36062]: 129 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:53.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:53 smithi071 conmon[33687]: cluster 2021-11-20T07:36:52.346278+0000 mgr.smithi071.ynqmwn (mgr.14162) 129 2021-11-20T07:36:53.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:53 smithi071 conmon[33687]: : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:54.886 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T07:36:54.950 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T07:36:54.951 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2021-11-20T07:36:54.951 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2021-11-20T07:36:54.951 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122206 s, 858 MB/s 2021-11-20T07:36:54.952 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2021-11-20T07:36:55.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2021-11-20T07:36:55.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:55 smithi112 conmon[36062]: cluster 2021-11-20T07:36:54.346563+0000 mgr.smithi071.ynqmwn (mgr.14162) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:55.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:55 smithi071 conmon[33687]: cluster 2021-11-20T07:36:54.346563+0000 mgr.smithi071.ynqmwn (mgr.14162) 130 : cluster [DBG] 2021-11-20T07:36:55.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:55 smithi071 conmon[33687]: pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:56.345 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[47260]: debug 2021-11-20T07:36:56.196+0000 7fee0c402080 -1 osd.4 0 log_to_monitors {default=true} 2021-11-20T07:36:56.691 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[36062]: audit 2021-11-20T07:36:55.506531+0000 mon.smithi071 ( 2021-11-20T07:36:56.692 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[36062]: mon.0) 370 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:56.692 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.200418+0000 mon.smithi112 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T07:36:56.692 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.200775+0000 mon.smithi071 (mon.0) 371 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T07:36:56.692 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:56 smithi112 conmon[36062]: cluster 2021-11-20T07:36:56.346828+0000 mgr.smithi071.ynqmwn (mgr.14162) 131 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:56.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: audit 2021-11-20T07:36:55.506531+0000 mon.smithi071 ( 2021-11-20T07:36:56.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: mon.0) 370 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:56.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: audit 2021-11-20T07:36:56.200418+0000 mon.smithi112 (mon.1) 8 : audit [INF] 2021-11-20T07:36:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: from='osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T07:36:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: audit 2021-11-20T07:36:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: 2021-11 2021-11-20T07:36:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: -20T07:36:56.200775+0000 mon.smithi071 (mon. 2021-11-20T07:36:56.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: 0) 371 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T07:36:56.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: cluster 2021-11-20T07:36:56.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: 2021-11-20T07:36:56.346828+0000 2021-11-20T07:36:56.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 131 : cluster [DBG] 2021-11-20T07:36:56.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:56 smithi071 conmon[33687]: pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.347+0000 7f756709b700 1 -- 172.21.15.112:0/2415181290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 msgr2=0x7f75600f4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.347+0000 7f756709b700 1 --2- 172.21.15.112:0/2415181290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f75600f4fe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f755c004660 tx=0x7f755c009b30).stop 2021-11-20T07:36:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- 172.21.15.112:0/2415181290 shutdown_connections 2021-11-20T07:36:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 --2- 172.21.15.112:0/2415181290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 0x7f75600f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 --2- 172.21.15.112:0/2415181290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f75600f4fe0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- 172.21.15.112:0/2415181290 >> 172.21.15.112:0/2415181290 conn(0x7f75600f0320 msgr2=0x7f75600f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:36:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- 172.21.15.112:0/2415181290 shutdown_connections 2021-11-20T07:36:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- 172.21.15.112:0/2415181290 wait complete. 2021-11-20T07:36:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 Processor -- start 2021-11-20T07:36:57.353 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- start start 2021-11-20T07:36:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 0x7f7560107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560107860 con 0x7f75600f5e60 2021-11-20T07:36:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f756709b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560108070 con 0x7f75600f4bc0 2021-11-20T07:36:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f7566099700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f7566099700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:58192/0 (socket says 172.21.15.112:58192) 2021-11-20T07:36:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.348+0000 7f7566099700 1 -- 172.21.15.112:0/3718047027 learned_addr learned my addr 172.21.15.112:0/3718047027 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:36:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7565898700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 0x7f7560107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7566099700 1 -- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 msgr2=0x7f7560107260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:36:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7566099700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 0x7f7560107260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:36:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7566099700 1 -- 172.21.15.112:0/3718047027 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c005040 con 0x7f75600f4bc0 2021-11-20T07:36:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7566099700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f755c004e30 tx=0x7f755c009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755c018070 con 0x7f75600f4bc0 2021-11-20T07:36:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560108290 con 0x7f75600f4bc0 2021-11-20T07:36:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f755c013410 con 0x7f75600f4bc0 2021-11-20T07:36:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75601087f0 con 0x7f75600f4bc0 2021-11-20T07:36:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.349+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755c01de20 con 0x7f75600f4bc0 2021-11-20T07:36:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.350+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560059100 con 0x7f75600f4bc0 2021-11-20T07:36:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.350+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f755c01d420 con 0x7f75600f4bc0 2021-11-20T07:36:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.351+0000 7f7556ffd700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 0x7f754c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:36:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.351+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3824+0+0 (secure 0 0 0) 0x7f755c077520 con 0x7f75600f4bc0 2021-11-20T07:36:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.351+0000 7f7565898700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 0x7f754c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:36:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.351+0000 7f7565898700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 0x7f754c05e7c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7550007f60 tx=0x7f755000c040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:36:57.361 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.355+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f755c04a800 con 0x7f75600f4bc0 2021-11-20T07:36:57.545 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:36:57.543+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f75600fcac0 con 0x7f754c05c300 2021-11-20T07:36:57.785 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[47260]: debug 2021-11-20T07:36:57.515+0000 7fee02973700 -1 osd.4 0 waiting for initial osdmap 2021-11-20T07:36:57.785 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[47260]: debug 2021-11-20T07:36:57.522+0000 7fedfd0eb700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:36:57.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.510457+0000 mon.smithi071 (mon.0) 372 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T07:36:57.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: cluster 2021-11-20T07:36:56.510512+0000 mon.smithi071 (mon.0) 373 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T07:36:57.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.510616+0000 mon.smithi071 (mon.0) 374 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:57.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.512972+0000 mon.smithi112 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:57.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.513427+0000 mon.smithi071 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:57.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:57 smithi112 conmon[36062]: audit 2021-11-20T07:36:56.972989+0000 mon.smithi071 (mon.0) 376 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:36:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: :36:56.510457+0000 mon.smithi071 (mon.0) 372 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T07:36:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: cluster 2021-11-20T07:36:56.510512+0000 2021-11-20T07:36:57.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: mon.smithi071 (mon.0) 373 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T07:36:57.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: audit 2021 2021-11-20T07:36:57.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: -11-20T07:36:56.510616+0000 mon.smithi071 (mon.0) 374 : audit 2021-11-20T07:36:57.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: audit 2021-11-20T07:36:56.512972+0000 2021-11-20T07:36:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: mon.smithi112 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:36:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: 11-20T07:36:56.513427+0000 mon.smithi071 (mon.0) 375 : audit 2021-11-20T07:36:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:36:57.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: audit 2021-11-20T07:36:56.972989 2021-11-20T07:36:57.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:57 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 376 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.512224+0000 mon.smithi071 (mon.0) 377 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:36:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: cluster 2021-11-20T07:36:57.512335+0000 mon.smithi071 (mon.0) 378 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T07:36:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.512721+0000 mon.smithi071 (mon.0) 379 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.516977+0000 mon.smithi071 (mon.0) 380 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.545596+0000 mgr.smithi071.ynqmwn (mgr.14162) 132 : audit [DBG] from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.546420+0000 mon.smithi071 (mon.0) 381 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:57.547873+0000 mgr.smithi071.ynqmwn (mgr.14162) 133 : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.548130+0000 mon.smithi071 2021-11-20T07:36:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: (mon.0) 382 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: audit 2021-11-20T07:36:57.548817+0000 mon.smithi071 (mon.0) 383 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:58 smithi112 conmon[36062]: cluster 2021-11-20T07:36:58.347368+0000 mgr.smithi071.ynqmwn (mgr.14162) 134 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11-20T07:36:57.512224+0000 mon.smithi071 2021-11-20T07:36:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: (mon.0) 377 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:36:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: cluster 2021-11-20T07:36:57.512335+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 378 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T07:36:58.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11-20T07:36:58.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 2021-11-20T07:36:57.512721+0000 mon.smithi071 (mon.0) 379 : 2021-11-20T07:36:58.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:58.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021- 2021-11-20T07:36:58.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 11-20T07:36:57.516977+0000 mon.smithi071 (mon.0 2021-11-20T07:36:58.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: ) 380 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:58.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11-20T07:36:57.545596 2021-11-20T07:36:58.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 132 : audit [DBG] from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:36:58.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11-20T07:36:57.546420+0000 mon.smithi071 (mon.0 2021-11-20T07:36:58.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: ) 381 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T07:36:58.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: cephadm 2021-11-20T07:36:57.547873+0000 mgr.smithi071.ynqmwn (mgr.14162) 133 2021-11-20T07:36:58.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: : cephadm [INF] Applying service osd.None on host smithi112... 2021-11-20T07:36:58.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:36:58.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 20T07:36:57.548130+0000 mon.smithi071 (mon.0) 382 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T07:36:58.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: audit 2021-11-20T07:36:58.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 2021-11-20T07:36:57.548817+0000 mon.smithi071 (mon.0) 2021-11-20T07:36:58.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 383 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:36:58.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: cluster 2021-11-20T07:36: 2021-11-20T07:36:58.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: 58.347368+0000 mgr.smithi071.ynqmwn (mgr.14162) 134 : 2021-11-20T07:36:58.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:58 smithi071 conmon[33687]: cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T07:36:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: cluster 2021-11-20T07:36:57. 2021-11-20T07:36:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: 223519+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: cluster 2021-11-20T07:36:57.223727+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: cluster 2021-11-20T07:36:58.516108+0000 mon.smithi071 (mon.0) 384 : cluster [INF] osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] boot 2021-11-20T07:36:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: cluster 2021-11-20T07:36:58.516197+0000 mon.smithi071 (mon.0) 385 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T07:36:59.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: audit 2021-11-20T07:36:58.516768+0000 mon.smithi071 (mon.0) 386 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:59.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: cephadm 2021-11-20T07:36:59.044829+0000 mgr.smithi071.ynqmwn (mgr.14162) 135 : cephadm [INF] Detected new or changed devices on smithi112 2021-11-20T07:36:59.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.049930+0000 mon.smithi071 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:59.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.053526+0000 mon.smithi071 (mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:59.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:36:59 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.056673+0000 mon.smithi071 (mon.0) 389 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: cluster 2021-11-20T07:36:57.223519+0000 osd.4 (osd. 2021-11-20T07:36:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:36:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: cluster 2021-11-20T07:36:57.223727+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:36:59.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: cluster 2021-11-20T07:36:58.516108 2021-11-20T07:36:59.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 384 : cluster [INF] 2021-11-20T07:36:59.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: osd.4 [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] boot 2021-11-20T07:36:59.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: cluster 2021-11-20T07:36:58.516197+0000 2021-11-20T07:36:59.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: mon.smithi071 (mon.0) 385 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T07:36:59.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:36:59.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: audit 2021-11-20T07:36:59.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:36:58.516768+0000 mon.smithi071 2021-11-20T07:36:59.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: (mon.0) 386 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T07:36:59.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: cephadm 2021- 2021-11-20T07:36:59.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 11-20T07:36:59.044829+0000 2021-11-20T07:36:59.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 135 : cephadm [INF] 2021-11-20T07:36:59.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: Detected new or changed devices on smithi112 2021-11-20T07:36:59.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: audit 2021-11-20T07:36:59.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:36:59.049930+0000 2021-11-20T07:36:59.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: mon.smithi071 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:59.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:36:59.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: audit 2021-11-20T07:36:59.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:36:59.053526+0000 mon.smithi071 ( 2021-11-20T07:36:59.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:36:59.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:36:59.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: -20T07:36:59.056673+0000 mon.smithi071 2021-11-20T07:36:59.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: (mon.0) 389 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:36:59.825 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:36:59 smithi071 conmon[33687]: 2021-11-20T07:37:00.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: audit 2021-11-20T07:36:59.847047 2021-11-20T07:37:00.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: +0000 mon.smithi112 (mon.1) 10 : audit [INF] from='client.? 172.21.15.112:0/1981361852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]: dispatch 2021-11-20T07:37:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: audit 2021-11-20T07:36:59.847430+0000 mon.smithi071 ( 2021-11-20T07:37:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: mon.0) 390 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]: dispatch 2021-11-20T07:37:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: audit 2021-11-20T07:37:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: 2021-11- 2021-11-20T07:37:00.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: 20T07:36:59.853732+0000 mon.smithi071 2021-11-20T07:37:00.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: (mon.0) 391 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]': finished 2021-11-20T07:37:00.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:37:00.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: -11-20T07:36:59.853786+0000 2021-11-20T07:37:00.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: mon.smithi071 (mon.0) 392 : cluster [DBG] 2021-11-20T07:37:00.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: osdmap e32: 6 total, 5 up, 6 in 2021-11-20T07:37:00.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: audit 2021-11-20T07:36: 2021-11-20T07:37:00.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: 59.853883+0000 mon.smithi071 (mon.0) 393 2021-11-20T07:37:00.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:00.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:37:00.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: :37:00.266604+0000 mon.smithi112 (mon. 2021-11-20T07:37:00.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: 1) 11 : audit [DBG] from='client.? 172.21.15.112:0/107354619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:37:00.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:37:00.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: -11-20T07:37:00.347681 2021-11-20T07:37:00.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 136 : cluster [DBG] 2021-11-20T07:37:00.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:00 smithi071 conmon[33687]: pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:01.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.847047+0000 mon.smithi112 2021-11-20T07:37:01.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: (mon.1) 10 : audit [INF] from='client.? 172.21.15.112:0/1981361852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]: dispatch 2021-11-20T07:37:01.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.847430+0000 mon.smithi071 (mon.0) 390 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]: dispatch 2021-11-20T07:37:01.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.853732+0000 mon.smithi071 (mon.0) 391 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfbcc628-860f-4297-85e1-58064adc31c5"}]': finished 2021-11-20T07:37:01.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: cluster 2021-11-20T07:36:59.853786+0000 mon.smithi071 (mon.0) 392 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2021-11-20T07:37:01.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: audit 2021-11-20T07:36:59.853883+0000 mon.smithi071 (mon.0) 393 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:01.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: audit 2021-11-20T07:37:00.266604+0000 mon.smithi112 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.112:0/107354619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T07:37:01.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:00 smithi112 conmon[36062]: cluster 2021-11-20T07:37:00.347681+0000 mgr.smithi071.ynqmwn (mgr.14162) 136 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:02.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:02 smithi112 conmon[36062]: cluster 2021-11-20T07:37:02.347938+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:37:02.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:02 smithi112 conmon[36062]: 137 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:02 smithi071 conmon[33687]: cluster 2021-11-20T07:37:02.347938+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:37:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:02 smithi071 conmon[33687]: .14162) 137 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:04.980 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:04 smithi112 conmon[36062]: audit 2021-11-20T07:37:03.756114+0000 mon.smithi071 (mon.0) 394 2021-11-20T07:37:04.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:04 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:04.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:04 smithi112 conmon[36062]: audit 2021-11-20T07:37:03.759059+0000 mon.smithi071 (mon.0) 395 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:04.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:04 smithi112 conmon[36062]: cluster 2021-11-20T07:37:04.348389+0000 mgr.smithi071.ynqmwn (mgr.14162) 138 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:05.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:04 smithi071 conmon[33687]: audit 2021-11-20T07:37:03.756114+0000 mon.smithi071 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:05.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:04 smithi071 conmon[33687]: audit 2021-11-20T07:37:03.759059+0000 mon.smithi071 (mon.0) 395 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:05.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:04 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:37:05.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:04 smithi071 conmon[33687]: 11-20T07:37:04.348389+0000 mgr.smithi071.ynqmwn (mgr.14162) 138 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:06.631 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:06 smithi112 conmon[36062]: audit 2021-11-20T07:37:06.040531+0000 mon.smithi071 (mon.0) 396 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T07:37:06.632 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:06 smithi112 conmon[36062]: audit 2021-11-20T07:37:06.041417+0000 mon.smithi071 (mon.0) 397 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:37:06.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:06 smithi071 conmon[33687]: audit 2021-11-20T07:37:06.040531+0000 mon.smithi071 (mon.0) 396 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T07:37:06.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:06 smithi071 conmon[33687]: audit 2021-11-20T07:37:06.041417+0000 mon.smithi071 (mon.0) 397 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:37:07.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:07 smithi112 conmon[36062]: cephadm 2021-11-20T07:37:06.042072+0000 mgr.smithi071.ynqmwn (mgr.14162) 139 : cephadm [INF] Deploying daemon osd.5 on smithi112 2021-11-20T07:37:07.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:07 smithi112 conmon[36062]: cluster 2021-11-20T07:37:06.348726+0000 mgr.smithi071.ynqmwn (mgr.14162) 140 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:07 smithi071 conmon[33687]: cephadm 2021-11-20T07:37:06.042072+0000 mgr.smithi071.ynqmwn (mgr.14162) 139 : cephadm [INF] Deploying daemon osd.5 on smithi112 2021-11-20T07:37:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:07 smithi071 conmon[33687]: cluster 2021-11-20T07:37:06.348726+0000 mgr.smithi071.ynqmwn (mgr.14162) 140 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:09.229 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 5 on host 'smithi112' 2021-11-20T07:37:09.231 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.227+0000 7f7556ffd700 1 -- 172.21.15.112:0/3718047027 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75600fcac0 con 0x7f754c05c300 2021-11-20T07:37:09.231 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 msgr2=0x7f754c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:09.231 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 0x7f754c05e7c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7550007f60 tx=0x7f755000c040).stop 2021-11-20T07:37:09.231 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 msgr2=0x7f7560106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:09.232 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f755c004e30 tx=0x7f755c009b30).stop 2021-11-20T07:37:09.232 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 shutdown_connections 2021-11-20T07:37:09.232 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75600f4bc0 0x7f7560106d70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:09.232 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f754c05c300 0x7f754c05e7c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:09.232 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 --2- 172.21.15.112:0/3718047027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75600f5e60 0x7f7560107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:09.233 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 >> 172.21.15.112:0/3718047027 conn(0x7f75600f0320 msgr2=0x7f75600f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:09.233 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 shutdown_connections 2021-11-20T07:37:09.233 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:09.229+0000 7f756709b700 1 -- 172.21.15.112:0/3718047027 wait complete. 2021-11-20T07:37:09.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:09 smithi112 conmon[36062]: cluster 2021-11-20T07:37:08.349304+0000 2021-11-20T07:37:09.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:09 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 141 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:09.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:09 smithi112 conmon[36062]: audit 2021-11-20T07:37:09.216279+0000 mon.smithi071 (mon.0) 398 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:37:09.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:09 smithi112 conmon[36062]: audit 2021-11-20T07:37:09.220866+0000 mon.smithi071 (mon.0) 399 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:09.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:09 smithi112 conmon[36062]: audit 2021-11-20T07:37:09.226745+0000 mon.smithi071 (mon.0) 400 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:09.620 DEBUG:teuthology.orchestra.run.smithi112:osd.5> sudo journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.5.service 2021-11-20T07:37:09.624 INFO:tasks.cephadm:Setting up client nodes... 2021-11-20T07:37:09.625 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T07:37:09.657 INFO:journalctl@ceph.osd.5.smithi112.stdout:-- Logs begin at Sat 2021-11-20 07:22:39 UTC. -- 2021-11-20T07:37:09.665 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: cluster 2021-11-20T07:37:08.349304+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:37:09.665 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: 14162) 141 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:09.665 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: audit 2021-11-20T07:37:09.216279+0000 mon.smithi071 (mon.0) 398 2021-11-20T07:37:09.665 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:37:09.666 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: audit 2021-11-20T07:37:09.666 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: 2021-11-20T07:37:09. 2021-11-20T07:37:09.666 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: 220866+0000 mon.smithi071 (mon.0) 399 : audit 2021-11-20T07:37:09.666 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:09.667 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: audit 2021-11-20T07:37:09.667 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: 2021-11-20T07 2021-11-20T07:37:09.667 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: :37:09.226745+0000 mon.smithi071 (mon.0 2021-11-20T07:37:09.667 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:09 smithi071 conmon[33687]: ) 400 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:11.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- 172.21.15.71:0/3039408994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 msgr2=0x7f9e2c0f7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:11.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3039408994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c0f7dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9e14004660 tx=0x7f9e14009b30).stop 2021-11-20T07:37:11.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- 172.21.15.71:0/3039408994 shutdown_connections 2021-11-20T07:37:11.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3039408994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c0f7dd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3039408994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 0x7f9e2c0f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- 172.21.15.71:0/3039408994 >> 172.21.15.71:0/3039408994 conn(0x7f9e2c0f1e10 msgr2=0x7f9e2c0f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:11.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- 172.21.15.71:0/3039408994 shutdown_connections 2021-11-20T07:37:11.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- 172.21.15.71:0/3039408994 wait complete. 2021-11-20T07:37:11.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 Processor -- start 2021-11-20T07:37:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.466+0000 7f9e318ad700 1 -- start start 2021-11-20T07:37:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 0x7f9e2c106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c1079a0 con 0x7f9e2c0f7970 2021-11-20T07:37:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c107ae0 con 0x7f9e2c0f66d0 2021-11-20T07:37:11.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:11.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47516/0 (socket says 172.21.15.71:47516) 2021-11-20T07:37:11.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 -- 172.21.15.71:0/3222897724 learned_addr learned my addr 172.21.15.71:0/3222897724 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:11.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e2affd700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 0x7f9e2c106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 -- 172.21.15.71:0/3222897724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 msgr2=0x7f9e2c106f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 0x7f9e2c106f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 -- 172.21.15.71:0/3222897724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14005040 con 0x7f9e2c0f7970 2021-11-20T07:37:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e23fff700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9e14000c00 tx=0x7f9e1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e14019070 con 0x7f9e2c0f7970 2021-11-20T07:37:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e14007e70 con 0x7f9e2c0f7970 2021-11-20T07:37:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e14014c90 con 0x7f9e2c0f7970 2021-11-20T07:37:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c108420 con 0x7f9e2c0f7970 2021-11-20T07:37:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.467+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c108980 con 0x7f9e2c0f7970 2021-11-20T07:37:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.468+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c045ad0 con 0x7f9e2c0f7970 2021-11-20T07:37:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.469+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e14025070 con 0x7f9e2c0f7970 2021-11-20T07:37:11.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.469+0000 7f9e237fe700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 0x7f9e1805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:11.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.469+0000 7f9e2affd700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 0x7f9e1805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:11.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.469+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f9e140769b0 con 0x7f9e2c0f7970 2021-11-20T07:37:11.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.470+0000 7f9e2affd700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 0x7f9e1805e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9e1c009fd0 tx=0x7f9e1c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:11.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.473+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e14049b70 con 0x7f9e2c0f7970 2021-11-20T07:37:11.692 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.690+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9e2c027a00 con 0x7f9e2c0f7970 2021-11-20T07:37:11.692 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:11 smithi071 conmon[33687]: cluster 2021-11-20T07:37:10.349619+0000 mgr.smithi071.ynqmwn (mgr.14162) 142 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:11.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.694+0000 7f9e237fe700 1 -- 172.21.15.71:0/3222897724 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9e1404d190 con 0x7f9e2c0f7970 2021-11-20T07:37:11.696 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2021-11-20T07:37:11.696 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCnpZhhxpFFKRAA+60lejt2ZmLxz3fTF+aajQ== 2021-11-20T07:37:11.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 msgr2=0x7f9e1805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:11.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 0x7f9e1805e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9e1c009fd0 tx=0x7f9e1c00b040).stop 2021-11-20T07:37:11.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 msgr2=0x7f9e2c1073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:11.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9e14000c00 tx=0x7f9e1400b040).stop 2021-11-20T07:37:11.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 shutdown_connections 2021-11-20T07:37:11.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e2c0f66d0 0x7f9e2c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e1805c2b0 0x7f9e1805e770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 --2- 172.21.15.71:0/3222897724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e2c0f7970 0x7f9e2c1073f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:11.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 >> 172.21.15.71:0/3222897724 conn(0x7f9e2c0f1e10 msgr2=0x7f9e2c0fb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:11.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 shutdown_connections 2021-11-20T07:37:11.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:11.696+0000 7f9e318ad700 1 -- 172.21.15.71:0/3222897724 wait complete. 2021-11-20T07:37:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:11 smithi112 conmon[36062]: cluster 2021-11-20T07:37:10.349619+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:37:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:11 smithi112 conmon[36062]: 142 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:12.058 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2021-11-20T07:37:12.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-11-20T07:37:12.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-11-20T07:37:12.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T07:37:12.414 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:12 smithi112 conmon[36062]: audit 2021-11-20T07:37:11.692270+0000 mon.smithi071 (mon.0) 401 : audit [INF] from='client.? 172.21.15.71:0/3222897724' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:12.415 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:12 smithi112 conmon[36062]: audit 2021-11-20T07:37:11.695658+0000 mon.smithi071 (mon.0) 402 : audit [INF] from='client.? 172.21.15.71:0/3222897724' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T07:37:12.415 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:12 smithi112 conmon[36062]: audit 2021-11-20T07:37:11.893704+0000 mon.smithi071 (mon.0) 403 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:12 smithi071 conmon[33687]: audit 2021-11-20T07:37:11.692270+0000 mon.smithi071 ( 2021-11-20T07:37:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:12 smithi071 conmon[33687]: mon.0) 401 : audit [INF] from='client.? 172.21.15.71:0/3222897724' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:12 smithi071 conmon[33687]: audit 2021-11-20T07:37:11.695658+0000 mon.smithi071 (mon.0) 402 : audit [INF] from='client.? 172.21.15.71:0/3222897724' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T07:37:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:12 smithi071 conmon[33687]: audit 2021-11-20T07:37:11.893704+0000 mon.smithi071 (mon.0) 403 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:13.352 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:37:13 smithi112 conmon[52158]: debug 2021-11-20T07:37:13.169+0000 7f367cac8080 -1 osd.5 0 log_to_monitors {default=true} 2021-11-20T07:37:13.606 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:13 smithi112 conmon[36062]: cluster 2021-11-20T07:37:12.349932+0000 mgr.smithi071.ynqmwn (mgr.14162) 143 : cluster 2021-11-20T07:37:13.606 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:13 smithi112 conmon[36062]: [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:13.606 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:13 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.172707+0000 mon.smithi112 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T07:37:13.606 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:13 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.173071+0000 mon.smithi071 (mon.0) 404 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T07:37:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:13 smithi071 conmon[33687]: cluster 2021-11-20T07:37:12.349932+0000 mgr.smithi071.ynqmwn (mgr.14162) 143 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:13 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:13 smithi071 conmon[33687]: 2021-11-20T07:37:13.172707+0000 mon.smithi112 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T07:37:13.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:13 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.173071+0000 mon.smithi071 (mon.0) 404 : audit 2021-11-20T07:37:13.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:13 smithi071 conmon[33687]: [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T07:37:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.990+0000 7f92ac503700 1 -- 172.21.15.112:0/2130230575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f5ed0 msgr2=0x7f92a40f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.990+0000 7f92ac503700 1 --2- 172.21.15.112:0/2130230575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f5ed0 0x7f92a40f6900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9294004660 tx=0x7f9294009b30).stop 2021-11-20T07:37:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- 172.21.15.112:0/2130230575 shutdown_connections 2021-11-20T07:37:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 --2- 172.21.15.112:0/2130230575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f5ed0 0x7f92a40f6900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 --2- 172.21.15.112:0/2130230575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f55c0 0x7f92a40f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- 172.21.15.112:0/2130230575 >> 172.21.15.112:0/2130230575 conn(0x7f92a40f0de0 msgr2=0x7f92a40f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- 172.21.15.112:0/2130230575 shutdown_connections 2021-11-20T07:37:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- 172.21.15.112:0/2130230575 wait complete. 2021-11-20T07:37:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 Processor -- start 2021-11-20T07:37:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- start start 2021-11-20T07:37:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f5ed0 0x7f92a410c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40605c0 con 0x7f92a40f5ed0 2021-11-20T07:37:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92ac503700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4060700 con 0x7f92a40f55c0 2021-11-20T07:37:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92aa29f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92aa29f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:58266/0 (socket says 172.21.15.112:58266) 2021-11-20T07:37:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92aa29f700 1 -- 172.21.15.112:0/894497111 learned_addr learned my addr 172.21.15.112:0/894497111 (peer_addr_for_me v2:172.21.15.112:0/0) 2021-11-20T07:37:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.991+0000 7f92a9a9e700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f5ed0 0x7f92a410c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92aa29f700 1 -- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f5ed0 msgr2=0x7f92a410c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92aa29f700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f5ed0 0x7f92a410c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92aa29f700 1 -- 172.21.15.112:0/894497111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294005040 con 0x7f92a40f55c0 2021-11-20T07:37:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92aa29f700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f92a00026d0 tx=0x7f92a0009de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a00079c0 con 0x7f92a40f55c0 2021-11-20T07:37:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a410c780 con 0x7f92a40f55c0 2021-11-20T07:37:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a410cd10 con 0x7f92a40f55c0 2021-11-20T07:37:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92a000f040 con 0x7f92a40f55c0 2021-11-20T07:37:14.000 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.992+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a00148b0 con 0x7f92a40f55c0 2021-11-20T07:37:14.000 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.993+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c004fa0 con 0x7f92a40f55c0 2021-11-20T07:37:14.001 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.993+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f92a00090d0 con 0x7f92a40f55c0 2021-11-20T07:37:14.001 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.994+0000 7f929affd700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 0x7f929005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:14.001 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.994+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4256+0+0 (secure 0 0 0) 0x7f92a00758c0 con 0x7f92a40f55c0 2021-11-20T07:37:14.002 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.994+0000 7f92a9a9e700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 0x7f929005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:14.002 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.994+0000 7f92a9a9e700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 0x7f929005e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9294009f50 tx=0x7f9294009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:14.002 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:13.997+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92a0048a70 con 0x7f92a40f55c0 2021-11-20T07:37:14.210 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.208+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f928c005670 con 0x7f92a40f55c0 2021-11-20T07:37:14.214 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.214+0000 7f929affd700 1 -- 172.21.15.112:0/894497111 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f92a004c090 con 0x7f92a40f55c0 2021-11-20T07:37:14.215 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2021-11-20T07:37:14.215 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCqpZhhG3KJDBAAovJs/BI58F8CnDssZbSVNQ== 2021-11-20T07:37:14.217 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 msgr2=0x7f929005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:14.217 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 0x7f929005e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9294009f50 tx=0x7f9294009b30).stop 2021-11-20T07:37:14.217 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 msgr2=0x7f92a4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:14.218 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f92a00026d0 tx=0x7f92a0009de0).stop 2021-11-20T07:37:14.218 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 shutdown_connections 2021-11-20T07:37:14.218 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92a40f55c0 0x7f92a4109cd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:14.219 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f929005c2b0 0x7f929005e770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:14.219 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 --2- 172.21.15.112:0/894497111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92a40f5ed0 0x7f92a410c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:14.219 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 >> 172.21.15.112:0/894497111 conn(0x7f92a40f0de0 msgr2=0x7f92a40fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:14.219 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 shutdown_connections 2021-11-20T07:37:14.220 INFO:teuthology.orchestra.run.smithi112.stderr:2021-11-20T07:37:14.216+0000 7f92ac503700 1 -- 172.21.15.112:0/894497111 wait complete. 2021-11-20T07:37:14.592 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2021-11-20T07:37:14.592 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-11-20T07:37:14.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-11-20T07:37:14.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T07:37:15.033 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[52158]: debug 2021-11-20T07:37:14.901+0000 7f367483c700 -1 osd.5 0 waiting for initial osdmap 2021-11-20T07:37:15.034 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[52158]: debug 2021-11-20T07:37:14.909+0000 7f366dfb2700 -1 osd.5 34 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T07:37:15.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.896469+0000 mon.smithi071 (mon.0) 405 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T07:37:15.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: cluster 2021-11-20T07:37:13.896524+0000 mon.smithi071 (mon.0) 406 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-20T07:37:15.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.896651+0000 mon.smithi071 (mon.0) 407 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:15.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.897014+0000 mon.smithi112 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:37:15.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:13.897328+0000 mon.smithi071 (mon.0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:37:15.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.209662+0000 mon.smithi112 (mon.1) 14 : audit [INF] from='client.? 172.21.15.112:0/894497111' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:15.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.210120+0000 mon.smithi071 (mon.0) 409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:15.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.214303+0000 mon.smithi071 (mon.0) 410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T07:37:15.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: cluster 2021-11-20T07:37:14.350252+0000 mgr.smithi071.ynqmwn (mgr.14162) 144 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:15.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:14 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.493484+0000 mon.smithi071 (mon.0) 411 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:15.065 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:15.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.896469+0000 mon.smithi071 (mon.0) 405 2021-11-20T07:37:15.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T07:37:15.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: cluster 2021-11-20T07:37:13.896524+0000 mon.smithi071 (mon.0) 406 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-20T07:37:15.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.896651+0000 mon.smithi071 (mon.0) 407 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:15.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.897014+0000 mon.smithi112 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:37:15.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:13.897328+0000 mon.smithi071 (mon.0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2021-11-20T07:37:15.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.209662+0000 mon.smithi112 (mon.1) 14 : audit [INF] from='client.? 172.21.15.112:0/894497111' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:15.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.210120+0000 mon.smithi071 (mon.0) 409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T07:37:15.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.214303+0000 mon.smithi071 (mon.0) 410 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T07:37:15.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: cluster 2021-11-20T07:37:14.350252+0000 mgr.smithi071.ynqmwn (mgr.14162) 144 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:15.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:14 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.493484+0000 mon.smithi071 (mon.0) 411 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:16.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.898374+0000 mon.smithi071 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:37:16.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: cluster 2021-11-20T07:37:14.898429+0000 mon.smithi071 (mon.0) 413 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T07:37:16.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.898718+0000 mon.smithi071 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:16.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:14.902132+0000 mon.smithi071 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:16.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: cephadm 2021-11-20T07:37:15.182988+0000 mgr.smithi071.ynqmwn (mgr.14162) 145 : cephadm [INF] Detected new or changed devices on smithi112 2021-11-20T07:37:16.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:15.187361+0000 mon.smithi071 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:16.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:15.542302+0000 mon.smithi071 (mon.0 2021-11-20T07:37:16.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: ) 417 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:16.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:15 smithi071 conmon[33687]: audit 2021-11-20T07:37:15.544786+0000 mon.smithi071 (mon.0) 418 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:37:16.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.898374+0000 mon.smithi071 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2021-11-20T07:37:16.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: cluster 2021-11-20T07:37:14.898429+0000 mon.smithi071 (mon.0) 413 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T07:37:16.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.898718+0000 mon.smithi071 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:16.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:14.902132+0000 mon.smithi071 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:16.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: cephadm 2021-11-20T07:37:15.182988+0000 mgr.smithi071.ynqmwn (mgr.14162) 145 : cephadm [INF] Detected new or changed devices on smithi112 2021-11-20T07:37:16.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:15.187361+0000 mon.smithi071 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:16.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:15.542302+0000 mon.smithi071 (mon.0) 417 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:16.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:15 smithi112 conmon[36062]: audit 2021-11-20T07:37:15.544786+0000 mon.smithi071 (mon.0) 418 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:37:16.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2991514995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 msgr2=0x7fb8780f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:16.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 --2- 172.21.15.71:0/2991514995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb8780f6dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb8680046e0 tx=0x7fb868009b30).stop 2021-11-20T07:37:16.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2991514995 shutdown_connections 2021-11-20T07:37:16.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 --2- 172.21.15.71:0/2991514995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb8780f6dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 --2- 172.21.15.71:0/2991514995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb8780f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2991514995 >> 172.21.15.71:0/2991514995 conn(0x7fb8780f0e20 msgr2=0x7fb8780f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:16.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2991514995 shutdown_connections 2021-11-20T07:37:16.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.389+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2991514995 wait complete. 2021-11-20T07:37:16.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 Processor -- start 2021-11-20T07:37:16.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 -- start start 2021-11-20T07:37:16.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb878109230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:16.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:16.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878109cd0 con 0x7fb8780f6960 2021-11-20T07:37:16.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87f5f1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878109e10 con 0x7fb8780f56c0 2021-11-20T07:37:16.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:16.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47530/0 (socket says 172.21.15.71:47530) 2021-11-20T07:37:16.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 -- 172.21.15.71:0/2297321544 learned_addr learned my addr 172.21.15.71:0/2297321544 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:16.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87d38d700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb878109230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:16.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 -- 172.21.15.71:0/2297321544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 msgr2=0x7fb878109230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:16.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb878109230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87cb8c700 1 -- 172.21.15.71:0/2297321544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868005040 con 0x7fb8780f6960 2021-11-20T07:37:16.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.390+0000 7fb87d38d700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb878109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:37:16.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb87cb8c700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb868000c00 tx=0x7fb868009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:16.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb868007ba0 con 0x7fb8780f6960 2021-11-20T07:37:16.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2297321544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87810a750 con 0x7fb8780f6960 2021-11-20T07:37:16.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb868007d00 con 0x7fb8780f6960 2021-11-20T07:37:16.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb868022610 con 0x7fb8780f6960 2021-11-20T07:37:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.391+0000 7fb87f5f1700 1 -- 172.21.15.71:0/2297321544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87810acb0 con 0x7fb8780f6960 2021-11-20T07:37:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.393+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878045ad0 con 0x7fb8780f6960 2021-11-20T07:37:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.394+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb86802a070 con 0x7fb8780f6960 2021-11-20T07:37:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.394+0000 7fb86dffb700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 0x7fb86405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.394+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb8680773f0 con 0x7fb8780f6960 2021-11-20T07:37:16.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.394+0000 7fb87d38d700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 0x7fb86405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.394+0000 7fb87d38d700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 0x7fb86405e760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb8780f67c0 tx=0x7fb874009c80).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.398+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb86804a3f0 con 0x7fb8780f6960 2021-11-20T07:37:16.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.562+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb87810b3b0 con 0x7fb8780f6960 2021-11-20T07:37:16.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.563+0000 7fb86dffb700 1 -- 172.21.15.71:0/2297321544 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+10836 (secure 0 0 0) 0x7fb86801e090 con 0x7fb8780f6960 2021-11-20T07:37:16.565 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:16.565 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":35,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:37:15.899596+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:37:16.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 msgr2=0x7fb86405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:16.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 0x7fb86405e760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb8780f67c0 tx=0x7fb874009c80).stop 2021-11-20T07:37:16.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 msgr2=0x7fb878109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:16.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb868000c00 tx=0x7fb868009b30).stop 2021-11-20T07:37:16.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 shutdown_connections 2021-11-20T07:37:16.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8780f56c0 0x7fb878109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb86405c2a0 0x7fb86405e760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 --2- 172.21.15.71:0/2297321544 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8780f6960 0x7fb878109720 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:16.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 >> 172.21.15.71:0/2297321544 conn(0x7fb8780f0e20 msgr2=0x7fb8780fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:16.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 shutdown_connections 2021-11-20T07:37:16.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:16.565+0000 7fb8637fe700 1 -- 172.21.15.71:0/2297321544 wait complete. 2021-11-20T07:37:17.066 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T07:36:35.321439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2021-11-20T07:37:17.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-11-20T07:37:17.082 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: cluster 2021-11-20T07:37 2021-11-20T07:37:17.083 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: :14.145250+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:37:17.083 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: cluster 2021-11-20T07:37:14.145357+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:37:17.083 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: cluster 2021-11-20T07:37:15.903460+0000 mon.smithi071 (mon.0) 419 : cluster [INF] osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] boot 2021-11-20T07:37:17.083 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: cluster 2021-11-20T07:37:15.903507+0000 mon.smithi071 (mon.0) 420 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2021-11-20T07:37:17.084 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: audit 2021-11-20T07:37:15.903839+0000 mon.smithi071 (mon.0) 421 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:17.084 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: cluster 2021-11-20T07:37:16.350607+0000 mgr.smithi071.ynqmwn (mgr.14162) 146 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:17.084 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: audit 2021-11-20T07:37:16.564007+0000 mon.smithi071 (mon.0) 422 : audit [DBG] from='client.? 172.21.15.71:0/2297321544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:17.084 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:16 smithi071 conmon[33687]: 2021-11-20T07:37:17.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: cluster 2021-11-20T07:37:14.145250+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T07:37:17.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: cluster 2021-11-20T07:37:14.145357+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T07:37:17.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: cluster 2021-11-20T07:37:15.903460+0000 mon.smithi071 (mon.0) 419 : cluster [INF] osd.5 [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] boot 2021-11-20T07:37:17.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: cluster 2021-11-20T07:37:17.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: 2021-11-20T07:37:15.903507+0000 mon.smithi071 (mon.0) 420 : cluster [DBG] 2021-11-20T07:37:17.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: osdmap e35: 6 total, 6 up, 6 in 2021-11-20T07:37:17.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:37:17.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: 07:37:15.903839+0000 mon.smithi071 (mon. 2021-11-20T07:37:17.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: 0) 421 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T07:37:17.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:37:17.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: -20T07:37:16.350607+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:37:17.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: mgr.14162) 146 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T07:37:17.287 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:37:17.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: 20T07:37:16.564007+0000 mon.smithi071 (mon. 2021-11-20T07:37:17.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:16 smithi112 conmon[36062]: 0) 422 : audit [DBG] from='client.? 172.21.15.71:0/2297321544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:17.513 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:18.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:17 smithi071 conmon[33687]: cluster 2021-11-20T07:37:16.917539+0000 mon.smithi071 (mon.0) 423 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T07:37:18.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:17 smithi112 conmon[36062]: cluster 2021-11-20T07:37:16.917539+0000 mon.smithi071 (mon.0) 423 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T07:37:18.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- 172.21.15.71:0/2022209590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f4870 msgr2=0x7fac080f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/2022209590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f4870 0x7fac080f4cf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fabf8004660 tx=0x7fabf8009b30).stop 2021-11-20T07:37:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- 172.21.15.71:0/2022209590 shutdown_connections 2021-11-20T07:37:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/2022209590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f4870 0x7fac080f4cf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:18.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/2022209590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f5b20 0x7fac080f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:18.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- 172.21.15.71:0/2022209590 >> 172.21.15.71:0/2022209590 conn(0x7fac080f0590 msgr2=0x7fac080f29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:18.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- 172.21.15.71:0/2022209590 shutdown_connections 2021-11-20T07:37:18.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- 172.21.15.71:0/2022209590 wait complete. 2021-11-20T07:37:18.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 Processor -- start 2021-11-20T07:37:18.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.912+0000 7fac0eb9b700 1 -- start start 2021-11-20T07:37:18.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0eb9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 0x7fac0810d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:18.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0eb9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:18.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0eb9b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0810e200 con 0x7fac080f5b20 2021-11-20T07:37:18.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0eb9b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0810eab0 con 0x7fac080f4870 2021-11-20T07:37:18.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:18.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47550/0 (socket says 172.21.15.71:47550) 2021-11-20T07:37:18.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 -- 172.21.15.71:0/471075303 learned_addr learned my addr 172.21.15.71:0/471075303 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:18.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0c937700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 0x7fac0810d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 -- 172.21.15.71:0/471075303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 msgr2=0x7fac0810d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 0x7fac0810d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 -- 172.21.15.71:0/471075303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8005040 con 0x7fac080f5b20 2021-11-20T07:37:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fac0c937700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 0x7fac0810d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:37:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.913+0000 7fabfffff700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fabf8009830 tx=0x7fabf8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:18.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.914+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf8018070 con 0x7fac080f5b20 2021-11-20T07:37:18.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.914+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0810ed30 con 0x7fac080f5b20 2021-11-20T07:37:18.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.914+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabf8013410 con 0x7fac080f5b20 2021-11-20T07:37:18.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.914+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf801c620 con 0x7fac080f5b20 2021-11-20T07:37:18.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.914+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0810f290 con 0x7fac080f5b20 2021-11-20T07:37:18.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.915+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08059100 con 0x7fac080f5b20 2021-11-20T07:37:18.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.915+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fabf801c8e0 con 0x7fac080f5b20 2021-11-20T07:37:18.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.916+0000 7fabfd7fa700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 0x7fabf005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:18.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.916+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fabf8077e50 con 0x7fac080f5b20 2021-11-20T07:37:18.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.916+0000 7fac0c937700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 0x7fabf005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:18.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.916+0000 7fac0c937700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 0x7fabf005e760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fac04001bf0 tx=0x7fac040012e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:18.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:18.919+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabf804ae60 con 0x7fac080f5b20 2021-11-20T07:37:19.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.074+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fac08027a00 con 0x7fac080f5b20 2021-11-20T07:37:19.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.074+0000 7fabfd7fa700 1 -- 172.21.15.71:0/471075303 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7fabf804e480 con 0x7fac080f5b20 2021-11-20T07:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2021-11-20T07:37:19.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 msgr2=0x7fabf005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:19.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 0x7fabf005e760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fac04001bf0 tx=0x7fac040012e0).stop 2021-11-20T07:37:19.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 msgr2=0x7fac0810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:19.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fabf8009830 tx=0x7fabf8009b30).stop 2021-11-20T07:37:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 shutdown_connections 2021-11-20T07:37:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac080f4870 0x7fac0810d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fabf005c2a0 0x7fabf005e760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 --2- 172.21.15.71:0/471075303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac080f5b20 0x7fac0810dbc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 >> 172.21.15.71:0/471075303 conn(0x7fac080f0590 msgr2=0x7fac080f92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 shutdown_connections 2021-11-20T07:37:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:19.076+0000 7fac0eb9b700 1 -- 172.21.15.71:0/471075303 wait complete. 2021-11-20T07:37:19.231 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:18 smithi071 conmon[33687]: cluster 2021-11-20T07:37:18.351037+0000 2021-11-20T07:37:19.232 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:18 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 147 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:19.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:18 smithi112 conmon[36062]: cluster 2021-11-20T07:37:18.351037+0000 mgr.smithi071.ynqmwn (mgr.14162) 147 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:19.467 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-11-20T07:37:19.468 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-11-20T07:37:19.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph mgr dump --format=json 2021-11-20T07:37:20.068 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:20.193 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:19 smithi071 conmon[33687]: audit 2021-11-20T07:37:18.931950+0000 2021-11-20T07:37:20.194 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:19 smithi071 conmon[33687]: mon.smithi071 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:37:20.195 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:19 smithi071 conmon[33687]: audit 2021-11-20T07:37:18.932184+0000 mon.smithi071 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:37:20.196 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:19 smithi071 conmon[33687]: audit 2021-11-20T07:37:19.075986+0000 mon.smithi071 (mon.0) 426 : audit [DBG] from='client.? 172.21.15.71:0/471075303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:37:20.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:19 smithi112 conmon[36062]: audit 2021-11-20T07:37:18.931950+0000 mon.smithi071 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:37:20.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:19 smithi112 conmon[36062]: 2021-11-20T07:37:20.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:19 smithi112 conmon[36062]: audit 2021-11-20T07:37:18.932184+0000 mon.smithi071 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:37:20.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:19 smithi112 conmon[36062]: audit 2021-11-20T07:37:19.075986+0000 mon.smithi071 2021-11-20T07:37:20.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:19 smithi112 conmon[36062]: (mon.0) 426 : audit [DBG] from='client.? 172.21.15.71:0/471075303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:37:21.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 -- 172.21.15.71:0/275813942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 msgr2=0x7fdd000f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 --2- 172.21.15.71:0/275813942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd000f6c00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fdcfc004660 tx=0x7fdcfc009b30).stop 2021-11-20T07:37:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 -- 172.21.15.71:0/275813942 shutdown_connections 2021-11-20T07:37:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 --2- 172.21.15.71:0/275813942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 0x7fdd000f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:21.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 --2- 172.21.15.71:0/275813942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd000f6c00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 -- 172.21.15.71:0/275813942 >> 172.21.15.71:0/275813942 conn(0x7fdd000f1f20 msgr2=0x7fdd000f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 -- 172.21.15.71:0/275813942 shutdown_connections 2021-11-20T07:37:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.395+0000 7fdd08239700 1 -- 172.21.15.71:0/275813942 wait complete. 2021-11-20T07:37:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 Processor -- start 2021-11-20T07:37:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 -- start start 2021-11-20T07:37:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 0x7fdd00069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0006a410 con 0x7fdd000f67e0 2021-11-20T07:37:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd08239700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0006a550 con 0x7fdd000f7a80 2021-11-20T07:37:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47558/0 (socket says 172.21.15.71:47558) 2021-11-20T07:37:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 -- 172.21.15.71:0/829643875 learned_addr learned my addr 172.21.15.71:0/829643875 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd057d4700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 0x7fdd00069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 -- 172.21.15.71:0/829643875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 msgr2=0x7fdd00069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 0x7fdd00069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 -- 172.21.15.71:0/829643875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc005040 con 0x7fdd000f67e0 2021-11-20T07:37:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.396+0000 7fdd05fd5700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fdcfc009f50 tx=0x7fdcfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.397+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc018070 con 0x7fdd000f67e0 2021-11-20T07:37:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.397+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcfc013410 con 0x7fdd000f67e0 2021-11-20T07:37:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.397+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0006a7d0 con 0x7fdd000f67e0 2021-11-20T07:37:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.397+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc01c3c0 con 0x7fdd000f67e0 2021-11-20T07:37:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.397+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0010f110 con 0x7fdd000f67e0 2021-11-20T07:37:21.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.398+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdcfc01d420 con 0x7fdd000f67e0 2021-11-20T07:37:21.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.398+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd00059100 con 0x7fdd000f67e0 2021-11-20T07:37:21.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.398+0000 7fdcf67fc700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 0x7fdcec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:21.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.398+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fdcfc077500 con 0x7fdd000f67e0 2021-11-20T07:37:21.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.398+0000 7fdd057d4700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 0x7fdcec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:21.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.399+0000 7fdd057d4700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 0x7fdcec05e760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fdcf0006fd0 tx=0x7fdcf0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:21.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.402+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcfc04a500 con 0x7fdd000f67e0 2021-11-20T07:37:21.598 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.597+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdd00045ad0 con 0x7fdd000f67e0 2021-11-20T07:37:21.601 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.599+0000 7fdcf67fc700 1 -- 172.21.15.71:0/829643875 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+143497 (secure 0 0 0) 0x7fdcfc04db20 con 0x7fdd000f67e0 2021-11-20T07:37:21.603 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:21.605 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 msgr2=0x7fdcec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:21.606 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 0x7fdcec05e760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fdcf0006fd0 tx=0x7fdcf0008040).stop 2021-11-20T07:37:21.606 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 msgr2=0x7fdd0006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:21.606 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fdcfc009f50 tx=0x7fdcfc009b30).stop 2021-11-20T07:37:21.607 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:21 smithi071 conmon[33687]: cluster 2021-11-20T07:37:20.351336+0000 mgr.smithi071.ynqmwn (mgr.14162) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:21.607 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 shutdown_connections 2021-11-20T07:37:21.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcec05c2a0 0x7fdcec05e760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:21.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd000f67e0 0x7fdd0006b9a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:21.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 --2- 172.21.15.71:0/829643875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd000f7a80 0x7fdd00069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:21.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.601+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 >> 172.21.15.71:0/829643875 conn(0x7fdd000f1f20 msgr2=0x7fdd000fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:21.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.602+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 shutdown_connections 2021-11-20T07:37:21.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:21.602+0000 7fdd08239700 1 -- 172.21.15.71:0/829643875 wait complete. 2021-11-20T07:37:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:21 smithi112 conmon[36062]: cluster 2021-11-20T07:37:20.351336+0000 mgr.smithi071.ynqmwn (mgr.14162) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:22.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":17,"active_gid":14162,"active_name":"smithi071.ynqmwn","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":429188327},{"type":"v1","addr":"172.21.15.71:6801","nonce":429188327}]},"active_addr":"172.21.15.71:6801/429188327","active_change":"2021-11-20T07:34:16.294371+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14194,"name":"smithi112.qaikrr","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi071.front.sepia.ceph.com:8443/","prometheus":"http://smithi071.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3120043174}]},{"addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3384899786}]}]}} 2021-11-20T07:37:22.021 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-11-20T07:37:22.022 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-11-20T07:37:22.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T07:37:22.455 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:22.833 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:22 smithi071 conmon[33687]: audit 2021-11-20T07:37:21.576770+0000 mon.smithi071 ( 2021-11-20T07:37:22.833 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:22 smithi071 conmon[33687]: mon.0) 427 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:22.834 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:22 smithi071 conmon[33687]: audit 2021-11-20T07:37:21.579821+0000 mon.smithi071 (mon.0) 428 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:22.834 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:22 smithi071 conmon[33687]: audit 2021-11-20T07:37:21.598474+0000 mon.smithi071 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.71:0/829643875' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T07:37:22.834 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:22 smithi071 conmon[33687]: cluster 2021-11-20T07:37:22.351625+0000 mgr.smithi071.ynqmwn (mgr.14162) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:23.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: audit 2021-11-20T07:37:21.576770+0000 mon.smithi071 (mon.0) 427 2021-11-20T07:37:23.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:23.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: audit 2021-11-20T07:37:21.579821+0000 mon.smithi071 (mon.0) 428 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:37:23.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: audit 2021-11-20T07:37:23.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: 2021-11-20T07:37:21.598474+0000 mon.smithi071 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.71:0/829643875' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T07:37:23.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:37:23.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: -20T07:37:22.351625+0000 mgr.smithi071.ynqmwn 2021-11-20T07:37:23.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:22 smithi112 conmon[36062]: (mgr.14162) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:23.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 -- 172.21.15.71:0/444500500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f5510 msgr2=0x7fc7800f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:23.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 --2- 172.21.15.71:0/444500500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f5510 0x7fc7800f5930 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc770004660 tx=0x7fc770009b30).stop 2021-11-20T07:37:23.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 -- 172.21.15.71:0/444500500 shutdown_connections 2021-11-20T07:37:23.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 --2- 172.21.15.71:0/444500500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f67b0 0x7fc7800f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.714 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 --2- 172.21.15.71:0/444500500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f5510 0x7fc7800f5930 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.714 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 -- 172.21.15.71:0/444500500 >> 172.21.15.71:0/444500500 conn(0x7fc7800f0c70 msgr2=0x7fc7800f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:23.715 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 -- 172.21.15.71:0/444500500 shutdown_connections 2021-11-20T07:37:23.715 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.706+0000 7fc77ffff700 1 -- 172.21.15.71:0/444500500 wait complete. 2021-11-20T07:37:23.715 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.707+0000 7fc77ffff700 1 Processor -- start 2021-11-20T07:37:23.715 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77ffff700 1 -- start start 2021-11-20T07:37:23.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f5510 0x7fc78010bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:23.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77ffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:23.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77ffff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780060870 con 0x7fc7800f67b0 2021-11-20T07:37:23.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77ffff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7800609b0 con 0x7fc7800f5510 2021-11-20T07:37:23.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:23.717 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47570/0 (socket says 172.21.15.71:47570) 2021-11-20T07:37:23.717 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 -- 172.21.15.71:0/4262156835 learned_addr learned my addr 172.21.15.71:0/4262156835 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:23.717 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77effd700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f5510 0x7fc78010bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:23.717 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 -- 172.21.15.71:0/4262156835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f5510 msgr2=0x7fc78010bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:23.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f5510 0x7fc78010bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 -- 172.21.15.71:0/4262156835 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc770005040 con 0x7fc7800f67b0 2021-11-20T07:37:23.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.708+0000 7fc77e7fc700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc77400c530 tx=0x7fc774007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:23.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.709+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc77400f650 con 0x7fc7800f67b0 2021-11-20T07:37:23.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.709+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc77400fc60 con 0x7fc7800f67b0 2021-11-20T07:37:23.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.709+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78010e980 con 0x7fc7800f67b0 2021-11-20T07:37:23.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.709+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc774012c30 con 0x7fc7800f67b0 2021-11-20T07:37:23.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.709+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78010ee60 con 0x7fc7800f67b0 2021-11-20T07:37:23.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.710+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc780045ad0 con 0x7fc7800f67b0 2021-11-20T07:37:23.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.710+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc774010070 con 0x7fc7800f67b0 2021-11-20T07:37:23.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.710+0000 7fc7677fe700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 0x7fc76805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:23.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.711+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc774075750 con 0x7fc7800f67b0 2021-11-20T07:37:23.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.711+0000 7fc77effd700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 0x7fc76805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:23.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.711+0000 7fc77effd700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 0x7fc76805e770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc780060600 tx=0x7fc770009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:23.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.714+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7740452b0 con 0x7fc7800f67b0 2021-11-20T07:37:23.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.863+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc780027a00 con 0x7fc7800f67b0 2021-11-20T07:37:23.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.864+0000 7fc7677fe700 1 -- 172.21.15.71:0/4262156835 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10859 (secure 0 0 0) 0x7fc7740487d0 con 0x7fc7800f67b0 2021-11-20T07:37:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:23.866 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":36,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:37:16.910019+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:37:14.145361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:37:23.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.865+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 msgr2=0x7fc76805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:23.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.865+0000 7fc77ffff700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 0x7fc76805e770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc780060600 tx=0x7fc770009b30).stop 2021-11-20T07:37:23.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.865+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 msgr2=0x7fc78010e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:23.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.865+0000 7fc77ffff700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc77400c530 tx=0x7fc774007e20).stop 2021-11-20T07:37:23.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 shutdown_connections 2021-11-20T07:37:23.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7800f5510 0x7fc78010bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc76805c2b0 0x7fc76805e770 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 --2- 172.21.15.71:0/4262156835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc7800f67b0 0x7fc78010e490 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:23.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 >> 172.21.15.71:0/4262156835 conn(0x7fc7800f0c70 msgr2=0x7fc7800fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:23.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 shutdown_connections 2021-11-20T07:37:23.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:23.866+0000 7fc77ffff700 1 -- 172.21.15.71:0/4262156835 wait complete. 2021-11-20T07:37:24.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:23 smithi071 conmon[33687]: audit 2021-11-20T07:37:23.865063+0000 mon.smithi071 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.71:0/4262156835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:24.268 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-11-20T07:37:24.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T07:37:24.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:23 smithi112 conmon[36062]: audit 2021-11-20T07:37:23.865063+0000 mon.smithi071 (mon.0) 430 : 2021-11-20T07:37:24.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:23 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/4262156835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:24.702 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:25.196 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:24 smithi071 conmon[33687]: cluster 2021-11-20T07:37:24.351937+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:37:25.196 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:24 smithi071 conmon[33687]: .14162) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:25.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:24 smithi112 conmon[36062]: cluster 2021-11-20T07:37:24.351937+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:37:25.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:24 smithi112 conmon[36062]: ) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:25.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 -- 172.21.15.71:0/3925797994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 msgr2=0x7f69300f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:25.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 --2- 172.21.15.71:0/3925797994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f69300f8100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f69200099e0 tx=0x7f6920004d10).stop 2021-11-20T07:37:25.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 -- 172.21.15.71:0/3925797994 shutdown_connections 2021-11-20T07:37:25.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 --2- 172.21.15.71:0/3925797994 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f69300f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:25.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 --2- 172.21.15.71:0/3925797994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f69300f8100 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:25.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 -- 172.21.15.71:0/3925797994 >> 172.21.15.71:0/3925797994 conn(0x7f69300f1d70 msgr2=0x7f69300f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:25.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.723+0000 7f693672d700 1 -- 172.21.15.71:0/3925797994 shutdown_connections 2021-11-20T07:37:25.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 -- 172.21.15.71:0/3925797994 wait complete. 2021-11-20T07:37:25.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 Processor -- start 2021-11-20T07:37:25.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 -- start start 2021-11-20T07:37:25.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f693006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:25.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:25.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693006a410 con 0x7f69300f7ce0 2021-11-20T07:37:25.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f693672d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693006a550 con 0x7f69300f62f0 2021-11-20T07:37:25.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f6934f2a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:25.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f6934f2a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47580/0 (socket says 172.21.15.71:47580) 2021-11-20T07:37:25.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.724+0000 7f6934f2a700 1 -- 172.21.15.71:0/1803344821 learned_addr learned my addr 172.21.15.71:0/1803344821 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:25.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f693572b700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f693006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:25.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f6934f2a700 1 -- 172.21.15.71:0/1803344821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 msgr2=0x7f693006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:25.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f6934f2a700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f693006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:25.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f6934f2a700 1 -- 172.21.15.71:0/1803344821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920009640 con 0x7f69300f7ce0 2021-11-20T07:37:25.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f693572b700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f693006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:37:25.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f6934f2a700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f692400b410 tx=0x7f6924006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:25.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69240125f0 con 0x7f69300f7ce0 2021-11-20T07:37:25.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693006a830 con 0x7f69300f7ce0 2021-11-20T07:37:25.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6924012c00 con 0x7f69300f7ce0 2021-11-20T07:37:25.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6924002aa0 con 0x7f69300f7ce0 2021-11-20T07:37:25.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.725+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693010f180 con 0x7f69300f7ce0 2021-11-20T07:37:25.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.726+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6930045ad0 con 0x7f69300f7ce0 2021-11-20T07:37:25.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.726+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f692400f030 con 0x7f69300f7ce0 2021-11-20T07:37:25.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.727+0000 7f692dffb700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 0x7f691805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:25.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.727+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6924075f10 con 0x7f69300f7ce0 2021-11-20T07:37:25.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.727+0000 7f693572b700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 0x7f691805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:25.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.727+0000 7f693572b700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 0x7f691805e770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f692000b620 tx=0x7f6920004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:25.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.731+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6924048f30 con 0x7f69300f7ce0 2021-11-20T07:37:25.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.881+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69300fef20 con 0x7f69300f7ce0 2021-11-20T07:37:25.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.882+0000 7f692dffb700 1 -- 172.21.15.71:0/1803344821 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10859 (secure 0 0 0) 0x7f692404c550 con 0x7f69300f7ce0 2021-11-20T07:37:25.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:25.885 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":36,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:37:16.910019+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:37:14.145361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:37:25.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 msgr2=0x7f691805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:25.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 0x7f691805e770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f692000b620 tx=0x7f6920004d10).stop 2021-11-20T07:37:25.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 msgr2=0x7f6930069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:25.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f692400b410 tx=0x7f6924006ef0).stop 2021-11-20T07:37:25.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 shutdown_connections 2021-11-20T07:37:25.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69300f62f0 0x7f693006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:25.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f691805c2b0 0x7f691805e770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:25.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 --2- 172.21.15.71:0/1803344821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69300f7ce0 0x7f6930069e20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:25.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 >> 172.21.15.71:0/1803344821 conn(0x7f69300f1d70 msgr2=0x7f69300fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:25.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 shutdown_connections 2021-11-20T07:37:25.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:25.884+0000 7f693672d700 1 -- 172.21.15.71:0/1803344821 wait complete. 2021-11-20T07:37:26.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-11-20T07:37:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:26 smithi071 conmon[33687]: audit 2021-11-20T07:37:25.883106+0000 mon.smithi071 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.71:0/1803344821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:26.735 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:26 smithi112 conmon[36062]: audit 2021-11-20T07:37:25.883106+0000 mon.smithi071 (mon.0) 431 2021-11-20T07:37:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:26 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/1803344821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:37:27.614 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:27 smithi071 conmon[33687]: cluster 2021-11-20T07:37:26.352320+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:37:27.617 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:27 smithi071 conmon[33687]: 14162) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:27.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:27 smithi112 conmon[36062]: cluster 2021-11-20T07:37:26.352320+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:37:27.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:27 smithi112 conmon[36062]: ) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:27.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 -- 172.21.15.71:0/531320828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f7690 msgr2=0x7f2cb00f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:27.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/531320828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f7690 0x7f2cb00f5ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2c9c004660 tx=0x7f2c9c009b30).stop 2021-11-20T07:37:27.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 -- 172.21.15.71:0/531320828 shutdown_connections 2021-11-20T07:37:27.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/531320828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f7690 0x7f2cb00f5ab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/531320828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb00f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 -- 172.21.15.71:0/531320828 >> 172.21.15.71:0/531320828 conn(0x7f2cb00f05e0 msgr2=0x7f2cb00f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:27.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 -- 172.21.15.71:0/531320828 shutdown_connections 2021-11-20T07:37:27.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.902+0000 7f2cb70c3700 1 -- 172.21.15.71:0/531320828 wait complete. 2021-11-20T07:37:27.908 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 Processor -- start 2021-11-20T07:37:27.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 -- start start 2021-11-20T07:37:27.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:27.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f7690 0x7f2cb010c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:27.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00f5780 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb70c3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00f58c0 con 0x7f2cb00f7690 2021-11-20T07:37:27.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:27.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47588/0 (socket says 172.21.15.71:47588) 2021-11-20T07:37:27.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 -- 172.21.15.71:0/3975845485 learned_addr learned my addr 172.21.15.71:0/3975845485 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:27.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2caffff700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f7690 0x7f2cb010c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:27.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 -- 172.21.15.71:0/3975845485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f7690 msgr2=0x7f2cb010c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:27.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f7690 0x7f2cb010c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.903+0000 7f2cb4e5f700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c005040 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cb4e5f700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2ca4002700 tx=0x7f2ca4009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca40079c0 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb010c860 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ca400f040 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca4014860 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.904+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb010cdf0 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.905+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2cb005dcf0 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.905+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2ca401b060 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.905+0000 7f2cad7fa700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:27.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.905+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2ca40759c0 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.905+0000 7f2caffff700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:27.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2cad7fa700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 0x7f2ca0064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:27.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ca0064b10 con 0x7f2ca0062030 2021-11-20T07:37:27.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca407a220 con 0x7f2cb00f6cf0 2021-11-20T07:37:27.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2cb5660700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 0x7f2ca0064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:27.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2caffff700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 0x7f2ca005e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2c9c009f50 tx=0x7f2c9c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:27.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.906+0000 7f2cb5660700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 0x7f2ca0064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:27.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.907+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== osd.0 v2:172.21.15.71:6802/1275932233 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f2ca0064b10 con 0x7f2ca0062030 2021-11-20T07:37:27.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 --> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2cb00581d0 con 0x7f2ca0062030 2021-11-20T07:37:27.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cad7fa700 1 -- 172.21.15.71:0/3975845485 <== osd.0 v2:172.21.15.71:6802/1275932233 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2cb00581d0 con 0x7f2ca0062030 2021-11-20T07:37:27.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 msgr2=0x7f2ca0064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:27.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 0x7f2ca0064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 msgr2=0x7f2ca005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:27.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 0x7f2ca005e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2c9c009f50 tx=0x7f2c9c009b30).stop 2021-11-20T07:37:27.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 msgr2=0x7f2cb0109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:27.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2ca4002700 tx=0x7f2ca4009d80).stop 2021-11-20T07:37:27.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 shutdown_connections 2021-11-20T07:37:27.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6802/1275932233,v1:172.21.15.71:6803/1275932233] conn(0x7f2ca0062030 0x7f2ca0064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cb00f6cf0 0x7f2cb0109d50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 --2- 172.21.15.71:0/3975845485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cb00f7690 0x7f2cb010c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:27.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 >> 172.21.15.71:0/3975845485 conn(0x7f2cb00f05e0 msgr2=0x7f2cb00fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:27.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 shutdown_connections 2021-11-20T07:37:27.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:27.921+0000 7f2cb70c3700 1 -- 172.21.15.71:0/3975845485 wait complete. 2021-11-20T07:37:28.331 INFO:teuthology.orchestra.run.smithi071.stdout:38654705686 2021-11-20T07:37:28.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-11-20T07:37:28.768 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:29.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:29 smithi071 conmon[33687]: cluster 2021-11-20T07:37:28.352609+0000 mgr.smithi071.ynqmwn (mgr.14162) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:29.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:29 smithi112 conmon[36062]: cluster 2021-11-20T07:37:28.352609+0000 mgr.smithi071.ynqmwn (mgr.14162) 152 : cluster [DBG] 2021-11-20T07:37:29.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:29 smithi112 conmon[36062]: pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:29.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 -- 172.21.15.71:0/3089370194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 msgr2=0x7f4e280f92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:29.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 --2- 172.21.15.71:0/3089370194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f92c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4e1c004660 tx=0x7f4e1c009b30).stop 2021-11-20T07:37:29.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 -- 172.21.15.71:0/3089370194 shutdown_connections 2021-11-20T07:37:29.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 --2- 172.21.15.71:0/3089370194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 0x7f4e280fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 --2- 172.21.15.71:0/3089370194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f92c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 -- 172.21.15.71:0/3089370194 >> 172.21.15.71:0/3089370194 conn(0x7f4e280f0ca0 msgr2=0x7f4e280f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:29.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 -- 172.21.15.71:0/3089370194 shutdown_connections 2021-11-20T07:37:29.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.943+0000 7f4e2d989700 1 -- 172.21.15.71:0/3089370194 wait complete. 2021-11-20T07:37:29.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 Processor -- start 2021-11-20T07:37:29.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 -- start start 2021-11-20T07:37:29.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:29.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 0x7f4e280f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:29.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280f5720 con 0x7f4e280607b0 2021-11-20T07:37:29.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2d989700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280f5860 con 0x7f4e280f97b0 2021-11-20T07:37:29.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2c987700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:29.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2c987700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47600/0 (socket says 172.21.15.71:47600) 2021-11-20T07:37:29.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.944+0000 7f4e2c987700 1 -- 172.21.15.71:0/1085432218 learned_addr learned my addr 172.21.15.71:0/1085432218 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:29.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e27fff700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 0x7f4e280f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:29.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2c987700 1 -- 172.21.15.71:0/1085432218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 msgr2=0x7f4e280f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:29.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2c987700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 0x7f4e280f51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2c987700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c005040 con 0x7f4e280607b0 2021-11-20T07:37:29.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2c987700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4e1c009830 tx=0x7f4e1c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:29.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e1c018070 con 0x7f4e280607b0 2021-11-20T07:37:29.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e280f5ae0 con 0x7f4e280607b0 2021-11-20T07:37:29.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e1c013410 con 0x7f4e280607b0 2021-11-20T07:37:29.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e1c01c620 con 0x7f4e280607b0 2021-11-20T07:37:29.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.945+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e280f6040 con 0x7f4e280607b0 2021-11-20T07:37:29.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.946+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e2805db60 con 0x7f4e280607b0 2021-11-20T07:37:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.946+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4e1c01c8e0 con 0x7f4e280607b0 2021-11-20T07:37:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.946+0000 7f4e257fa700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 0x7f4e1805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4e1c077a60 con 0x7f4e280607b0 2021-11-20T07:37:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e27fff700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 0x7f4e1805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e257fa700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 0x7f4e180644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e18064b20 con 0x7f4e18062040 2021-11-20T07:37:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e1c07c220 con 0x7f4e280607b0 2021-11-20T07:37:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e2d188700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 0x7f4e180644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e27fff700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 0x7f4e1805e770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4e140041d0 tx=0x7f4e14009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.947+0000 7f4e2d188700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 0x7f4e180644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.948+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== osd.1 v2:172.21.15.71:6810/12679585 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f4e18064b20 con 0x7f4e18062040 2021-11-20T07:37:29.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 --> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e28058150 con 0x7f4e18062040 2021-11-20T07:37:29.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e257fa700 1 -- 172.21.15.71:0/1085432218 <== osd.1 v2:172.21.15.71:6810/12679585 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e28058150 con 0x7f4e18062040 2021-11-20T07:37:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 msgr2=0x7f4e180644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 0x7f4e180644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 msgr2=0x7f4e1805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 0x7f4e1805e770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4e140041d0 tx=0x7f4e14009040).stop 2021-11-20T07:37:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 msgr2=0x7f4e280f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4e1c009830 tx=0x7f4e1c009b30).stop 2021-11-20T07:37:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 shutdown_connections 2021-11-20T07:37:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4e1805c2b0 0x7f4e1805e770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e280607b0 0x7f4e280f6d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.967 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.71:6810/12679585,v1:172.21.15.71:6811/12679585] conn(0x7f4e18062040 0x7f4e180644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.967 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 --2- 172.21.15.71:0/1085432218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e280f97b0 0x7f4e280f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:29.967 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 >> 172.21.15.71:0/1085432218 conn(0x7f4e280f0ca0 msgr2=0x7f4e280f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:29.967 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.962+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 shutdown_connections 2021-11-20T07:37:29.968 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:29.963+0000 7f4e2d989700 1 -- 172.21.15.71:0/1085432218 wait complete. 2021-11-20T07:37:30.365 INFO:teuthology.orchestra.run.smithi071.stdout:60129542163 2021-11-20T07:37:30.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-11-20T07:37:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:31.530 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:31 smithi071 conmon[33687]: cluster 2021-11-20T07:37:30.352887+0000 mgr.smithi071.ynqmwn (mgr.14162) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:31.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:31 smithi112 conmon[36062]: cluster 2021-11-20T07:37:30.352887+0000 mgr.smithi071.ynqmwn (mgr.14162) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:31.993 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 -- 172.21.15.71:0/4123491387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 msgr2=0x7f29e80f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:31.993 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 --2- 172.21.15.71:0/4123491387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e80f5940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f29e4004660 tx=0x7f29e4009b30).stop 2021-11-20T07:37:31.994 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 -- 172.21.15.71:0/4123491387 shutdown_connections 2021-11-20T07:37:31.994 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 --2- 172.21.15.71:0/4123491387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e80f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:31.994 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 --2- 172.21.15.71:0/4123491387 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e80f5940 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:31.994 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.989+0000 7f29ee90f700 1 -- 172.21.15.71:0/4123491387 >> 172.21.15.71:0/4123491387 conn(0x7f29e80f0ca0 msgr2=0x7f29e80f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:31.995 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 -- 172.21.15.71:0/4123491387 shutdown_connections 2021-11-20T07:37:31.995 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 -- 172.21.15.71:0/4123491387 wait complete. 2021-11-20T07:37:31.995 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 Processor -- start 2021-11-20T07:37:31.995 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 -- start start 2021-11-20T07:37:31.995 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:31.996 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e8109620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:31.996 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8109c60 con 0x7f29e80f5520 2021-11-20T07:37:31.996 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ee90f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e810a510 con 0x7f29e80f67c0 2021-11-20T07:37:31.996 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ed90d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:31.997 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ed90d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47612/0 (socket says 172.21.15.71:47612) 2021-11-20T07:37:31.997 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.990+0000 7f29ed90d700 1 -- 172.21.15.71:0/3717124335 learned_addr learned my addr 172.21.15.71:0/3717124335 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:31.997 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed10c700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e8109620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:31.997 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed90d700 1 -- 172.21.15.71:0/3717124335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 msgr2=0x7f29e8109620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:31.997 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed90d700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e8109620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:31.998 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed90d700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e4005040 con 0x7f29e80f5520 2021-11-20T07:37:31.998 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed10c700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:37:31.998 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ed90d700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29e4009f50 tx=0x7f29e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:31.998 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29e4018070 con 0x7f29e80f5520 2021-11-20T07:37:31.999 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29e4013410 con 0x7f29e80f5520 2021-11-20T07:37:31.999 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e810a790 con 0x7f29e80f5520 2021-11-20T07:37:31.999 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29e401c620 con 0x7f29e80f5520 2021-11-20T07:37:31.999 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.991+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e810acf0 con 0x7f29e80f5520 2021-11-20T07:37:31.999 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.992+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29e805db60 con 0x7f29e80f5520 2021-11-20T07:37:32.001 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.992+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f29e401d420 con 0x7f29e80f5520 2021-11-20T07:37:32.001 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.992+0000 7f29de7fc700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 0x7f29d405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:32.001 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f29e4077a40 con 0x7f29e80f5520 2021-11-20T07:37:32.002 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29ed10c700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 0x7f29d405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:32.002 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29de7fc700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 0x7f29d40644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:32.002 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29d4064b20 con 0x7f29d4062040 2021-11-20T07:37:32.002 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29e407c220 con 0x7f29e80f5520 2021-11-20T07:37:32.003 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29ee10e700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 0x7f29d40644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:32.003 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.993+0000 7f29ee10e700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 0x7f29d40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:32.003 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.994+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== osd.2 v2:172.21.15.71:6818/3317104048 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f29d4064b20 con 0x7f29d4062040 2021-11-20T07:37:32.003 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:31.994+0000 7f29ed10c700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 0x7f29d405e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f29d800b830 tx=0x7f29d8008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:32.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 --> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29e810b290 con 0x7f29d4062040 2021-11-20T07:37:32.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29de7fc700 1 -- 172.21.15.71:0/3717124335 <== osd.2 v2:172.21.15.71:6818/3317104048 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f29e810b290 con 0x7f29d4062040 2021-11-20T07:37:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 msgr2=0x7f29d40644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 0x7f29d40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 msgr2=0x7f29d405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 0x7f29d405e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f29d800b830 tx=0x7f29d8008040).stop 2021-11-20T07:37:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 msgr2=0x7f29e8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29e4009f50 tx=0x7f29e4009b30).stop 2021-11-20T07:37:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 shutdown_connections 2021-11-20T07:37:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6818/3317104048,v1:172.21.15.71:6819/3317104048] conn(0x7f29d4062040 0x7f29d40644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:32.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29d405c2b0 0x7f29d405e770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:32.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e80f5520 0x7f29e8109130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:32.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 --2- 172.21.15.71:0/3717124335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e80f67c0 0x7f29e8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:32.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.008+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 >> 172.21.15.71:0/3717124335 conn(0x7f29e80f0ca0 msgr2=0x7f29e80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:32.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.009+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 shutdown_connections 2021-11-20T07:37:32.014 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:32.009+0000 7f29ee90f700 1 -- 172.21.15.71:0/3717124335 wait complete. 2021-11-20T07:37:32.402 INFO:teuthology.orchestra.run.smithi071.stdout:77309411345 2021-11-20T07:37:32.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-11-20T07:37:32.839 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:33.546 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:33 smithi071 conmon[33687]: cluster 2021-11-20T07:37:32.353328+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:37:33.546 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:33 smithi071 conmon[33687]: 14162) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:33 smithi112 conmon[36062]: cluster 2021-11-20T07:37:32.353328+0000 mgr.smithi071.ynqmwn (mgr.14162) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:34.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.178+0000 7fc0f56ce700 1 -- 172.21.15.71:0/76575371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f56c0 msgr2=0x7fc0f00f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:34.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.178+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/76575371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f5ae0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc0e00099e0 tx=0x7fc0e0004d10).stop 2021-11-20T07:37:34.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 -- 172.21.15.71:0/76575371 shutdown_connections 2021-11-20T07:37:34.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/76575371 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/76575371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f5ae0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 -- 172.21.15.71:0/76575371 >> 172.21.15.71:0/76575371 conn(0x7fc0f00f0e20 msgr2=0x7fc0f00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:34.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 -- 172.21.15.71:0/76575371 shutdown_connections 2021-11-20T07:37:34.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 -- 172.21.15.71:0/76575371 wait complete. 2021-11-20T07:37:34.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 Processor -- start 2021-11-20T07:37:34.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.179+0000 7fc0f56ce700 1 -- start start 2021-11-20T07:37:34.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:34.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:34.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00f8200 con 0x7fc0f00f6960 2021-11-20T07:37:34.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00f8340 con 0x7fc0f00f56c0 2021-11-20T07:37:34.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:34.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47626/0 (socket says 172.21.15.71:47626) 2021-11-20T07:37:34.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 -- 172.21.15.71:0/688496108 learned_addr learned my addr 172.21.15.71:0/688496108 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:34.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0eeffd700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:34.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 -- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 msgr2=0x7fc0f00f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:34.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0009640 con 0x7fc0f00f6960 2021-11-20T07:37:34.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0eeffd700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:37:34.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0ee7fc700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc0e40026d0 tx=0x7fc0e4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:34.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e4009320 con 0x7fc0f00f6960 2021-11-20T07:37:34.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f00f8620 con 0x7fc0f00f6960 2021-11-20T07:37:34.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e400f040 con 0x7fc0f00f6960 2021-11-20T07:37:34.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e40147e0 con 0x7fc0f00f6960 2021-11-20T07:37:34.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.180+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f010f1f0 con 0x7fc0f00f6960 2021-11-20T07:37:34.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.181+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0f005dd30 con 0x7fc0f00f6960 2021-11-20T07:37:34.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.182+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc0e4009480 con 0x7fc0f00f6960 2021-11-20T07:37:34.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.182+0000 7fc0d77fe700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 0x7fc0d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:34.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.182+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc0e4076270 con 0x7fc0f00f6960 2021-11-20T07:37:34.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.182+0000 7fc0eeffd700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 0x7fc0d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:34.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0d77fe700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 0x7fc0d8064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:34.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0d8064b10 con 0x7fc0d8062030 2021-11-20T07:37:34.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0e40788e0 con 0x7fc0f00f6960 2021-11-20T07:37:34.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0eeffd700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 0x7fc0d805e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc0e0004750 tx=0x7fc0e0004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:34.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0ef7fe700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 0x7fc0d8064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:34.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.183+0000 7fc0ef7fe700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 0x7fc0d8064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:34.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.184+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== osd.3 v2:172.21.15.112:6800/3567947244 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fc0d8064b10 con 0x7fc0d8062030 2021-11-20T07:37:34.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 --> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0f00fafe0 con 0x7fc0d8062030 2021-11-20T07:37:34.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0d77fe700 1 -- 172.21.15.71:0/688496108 <== osd.3 v2:172.21.15.112:6800/3567947244 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc0f00fafe0 con 0x7fc0d8062030 2021-11-20T07:37:34.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 msgr2=0x7fc0d8064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:34.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 0x7fc0d8064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 msgr2=0x7fc0d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:34.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 0x7fc0d805e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc0e0004750 tx=0x7fc0e0004d10).stop 2021-11-20T07:37:34.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 msgr2=0x7fc0f00f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:34.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.198+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc0e40026d0 tx=0x7fc0e4009de0).stop 2021-11-20T07:37:34.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 shutdown_connections 2021-11-20T07:37:34.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f00f56c0 0x7fc0f00f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.112:6800/3567947244,v1:172.21.15.112:6801/3567947244] conn(0x7fc0d8062030 0x7fc0d8064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0d805c2a0 0x7fc0d805e760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 --2- 172.21.15.71:0/688496108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f00f6960 0x7fc0f00f7ca0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:34.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 >> 172.21.15.71:0/688496108 conn(0x7fc0f00f0e20 msgr2=0x7fc0f00603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:34.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 shutdown_connections 2021-11-20T07:37:34.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:34.199+0000 7fc0f56ce700 1 -- 172.21.15.71:0/688496108 wait complete. 2021-11-20T07:37:34.550 INFO:teuthology.orchestra.run.smithi071.stdout:107374182413 2021-11-20T07:37:34.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-11-20T07:37:34.990 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:35.486 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:35 smithi071 conmon[33687]: cluster 2021-11-20T07:37:34.353595+0000 mgr.smithi071.ynqmwn (mgr.14162) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:35 smithi112 conmon[36062]: cluster 2021-11-20T07:37:34.353595+0000 mgr.smithi071.ynqmwn (mgr.14162) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:36.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 -- 172.21.15.71:0/3733611330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 msgr2=0x7fe6680f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:36.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3733611330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680f5ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe664004660 tx=0x7fe664009b30).stop 2021-11-20T07:37:36.154 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 -- 172.21.15.71:0/3733611330 shutdown_connections 2021-11-20T07:37:36.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3733611330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3733611330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680f5ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 -- 172.21.15.71:0/3733611330 >> 172.21.15.71:0/3733611330 conn(0x7fe6680f0e20 msgr2=0x7fe6680f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:36.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 -- 172.21.15.71:0/3733611330 shutdown_connections 2021-11-20T07:37:36.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.147+0000 7fe66fc10700 1 -- 172.21.15.71:0/3733611330 wait complete. 2021-11-20T07:37:36.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 Processor -- start 2021-11-20T07:37:36.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 -- start start 2021-11-20T07:37:36.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:36.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:36.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680f8a20 con 0x7fe6680f56c0 2021-11-20T07:37:36.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680f8b60 con 0x7fe6680f6960 2021-11-20T07:37:36.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:36.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47636/0 (socket says 172.21.15.71:47636) 2021-11-20T07:37:36.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 -- 172.21.15.71:0/3005668166 learned_addr learned my addr 172.21.15.71:0/3005668166 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:36.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d1ab700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:36.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 -- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 msgr2=0x7fe6680f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:36.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664005040 con 0x7fe6680f56c0 2021-11-20T07:37:36.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d1ab700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:37:36.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66d9ac700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe664004e30 tx=0x7fe664009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:36.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe664018070 con 0x7fe6680f56c0 2021-11-20T07:37:36.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6680f8de0 con 0x7fe6680f56c0 2021-11-20T07:37:36.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.148+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe664013410 con 0x7fe6680f56c0 2021-11-20T07:37:36.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.149+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe66401c3c0 con 0x7fe6680f56c0 2021-11-20T07:37:36.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.149+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6680f9340 con 0x7fe6680f56c0 2021-11-20T07:37:36.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.149+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe66805dd30 con 0x7fe6680f56c0 2021-11-20T07:37:36.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe66401d420 con 0x7fe6680f56c0 2021-11-20T07:37:36.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 0x7fe65405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:36.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe664077ee0 con 0x7fe6680f56c0 2021-11-20T07:37:36.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe66d1ab700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 0x7fe65405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:36.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 0x7fe654064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:36.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe654064b10 con 0x7fe654062030 2021-11-20T07:37:36.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.150+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6640782e0 con 0x7fe6680f56c0 2021-11-20T07:37:36.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.151+0000 7fe66d1ab700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 0x7fe65405e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe658004b90 tx=0x7fe65800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:36.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.151+0000 7fe66e1ad700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 0x7fe654064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:36.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.151+0000 7fe66e1ad700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 0x7fe654064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:36.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.152+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== osd.4 v2:172.21.15.112:6808/297260 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fe654064b10 con 0x7fe654062030 2021-11-20T07:37:36.167 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.166+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 --> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6680604d0 con 0x7fe654062030 2021-11-20T07:37:36.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.166+0000 7fe65e7fc700 1 -- 172.21.15.71:0/3005668166 <== osd.4 v2:172.21.15.112:6808/297260 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe6680604d0 con 0x7fe654062030 2021-11-20T07:37:36.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 msgr2=0x7fe654064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:36.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 0x7fe654064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 msgr2=0x7fe65405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:36.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 0x7fe65405e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe658004b90 tx=0x7fe65800b040).stop 2021-11-20T07:37:36.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 msgr2=0x7fe6680fa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:36.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe664004e30 tx=0x7fe664009b30).stop 2021-11-20T07:37:36.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 shutdown_connections 2021-11-20T07:37:36.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe65405c2a0 0x7fe65405e760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6680f56c0 0x7fe6680fa040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:6808/297260,v1:172.21.15.112:6809/297260] conn(0x7fe654062030 0x7fe654064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 --2- 172.21.15.71:0/3005668166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6680f6960 0x7fe6680f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:36.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 >> 172.21.15.71:0/3005668166 conn(0x7fe6680f0e20 msgr2=0x7fe66805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:36.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 shutdown_connections 2021-11-20T07:37:36.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:36.167+0000 7fe66fc10700 1 -- 172.21.15.71:0/3005668166 wait complete. 2021-11-20T07:37:36.589 INFO:teuthology.orchestra.run.smithi071.stdout:133143986186 2021-11-20T07:37:36.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-11-20T07:37:37.025 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:37.404 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:37 smithi071 conmon[33687]: cluster 2021-11-20T07:37:36.353983+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:37:37.405 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:37 smithi071 conmon[33687]: mgr.14162) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:37 smithi112 conmon[36062]: cluster 2021-11-20T07:37:36.353983+0000 mgr.smithi071.ynqmwn (mgr.14162) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:38.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 -- 172.21.15.71:0/1900643852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 msgr2=0x7f2d100f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:38.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 --2- 172.21.15.71:0/1900643852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f4ec0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f2d000099e0 tx=0x7f2d00004d10).stop 2021-11-20T07:37:38.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 -- 172.21.15.71:0/1900643852 shutdown_connections 2021-11-20T07:37:38.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 --2- 172.21.15.71:0/1900643852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 --2- 172.21.15.71:0/1900643852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f4ec0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.175+0000 7f2d159ba700 1 -- 172.21.15.71:0/1900643852 >> 172.21.15.71:0/1900643852 conn(0x7f2d100f05e0 msgr2=0x7f2d100f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:38.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 -- 172.21.15.71:0/1900643852 shutdown_connections 2021-11-20T07:37:38.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 -- 172.21.15.71:0/1900643852 wait complete. 2021-11-20T07:37:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 Processor -- start 2021-11-20T07:37:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 -- start start 2021-11-20T07:37:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:38.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:38.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100f6030 con 0x7f2d100f4aa0 2021-11-20T07:37:38.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.176+0000 7f2d159ba700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100f6170 con 0x7f2d100f5440 2021-11-20T07:37:38.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:38.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47648/0 (socket says 172.21.15.71:47648) 2021-11-20T07:37:38.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 -- 172.21.15.71:0/4035619551 learned_addr learned my addr 172.21.15.71:0/4035619551 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:38.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0e7fc700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:38.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 -- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 msgr2=0x7f2d100f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:38.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009640 con 0x7f2d100f4aa0 2021-11-20T07:37:38.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0e7fc700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:37:38.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d0effd700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2d000078b0 tx=0x7f2d00004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:38.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d0000e030 con 0x7f2d100f4aa0 2021-11-20T07:37:38.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d00007e00 con 0x7f2d100f4aa0 2021-11-20T07:37:38.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d100f63f0 con 0x7f2d100f4aa0 2021-11-20T07:37:38.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d00018070 con 0x7f2d100f4aa0 2021-11-20T07:37:38.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.177+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d100f68d0 con 0x7f2d100f4aa0 2021-11-20T07:37:38.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.178+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d1005dcf0 con 0x7f2d100f4aa0 2021-11-20T07:37:38.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.178+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2d000137e0 con 0x7f2d100f4aa0 2021-11-20T07:37:38.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d077fe700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 0x7f2cfc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:38.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2d000779f0 con 0x7f2d100f4aa0 2021-11-20T07:37:38.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d0e7fc700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 0x7f2cfc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:38.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d077fe700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 0x7f2cfc064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:38.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cfc064b10 con 0x7f2cfc062030 2021-11-20T07:37:38.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d0007c220 con 0x7f2d100f4aa0 2021-11-20T07:37:38.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d0e7fc700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 0x7f2cfc05e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2d100f5880 tx=0x7f2cf8009c80).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:38.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d0f7fe700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 0x7f2cfc064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:38.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.179+0000 7f2d0f7fe700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 0x7f2cfc064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:38.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.180+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== osd.5 v2:172.21.15.112:6816/3514154095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f2cfc064b10 con 0x7f2cfc062030 2021-11-20T07:37:38.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.194+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 --> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2d100581d0 con 0x7f2cfc062030 2021-11-20T07:37:38.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d077fe700 1 -- 172.21.15.71:0/4035619551 <== osd.5 v2:172.21.15.112:6816/3514154095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2d100581d0 con 0x7f2cfc062030 2021-11-20T07:37:38.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 msgr2=0x7f2cfc064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:38.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 0x7f2cfc064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 msgr2=0x7f2cfc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:38.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 0x7f2cfc05e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2d100f5880 tx=0x7f2cf8009c80).stop 2021-11-20T07:37:38.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 msgr2=0x7f2d100f7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:38.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2d000078b0 tx=0x7f2d00004d10).stop 2021-11-20T07:37:38.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 shutdown_connections 2021-11-20T07:37:38.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:6816/3514154095,v1:172.21.15.112:6817/3514154095] conn(0x7f2cfc062030 0x7f2cfc064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2cfc05c2a0 0x7f2cfc05e760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d100f4aa0 0x7f2d100f7650 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 --2- 172.21.15.71:0/4035619551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d100f5440 0x7f2d100f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:38.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 >> 172.21.15.71:0/4035619551 conn(0x7f2d100f05e0 msgr2=0x7f2d100fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:38.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 shutdown_connections 2021-11-20T07:37:38.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:38.195+0000 7f2d159ba700 1 -- 172.21.15.71:0/4035619551 wait complete. 2021-11-20T07:37:38.521 INFO:teuthology.orchestra.run.smithi071.stdout:150323855367 2021-11-20T07:37:38.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-11-20T07:37:38.957 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:39.689 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:39 smithi071 conmon[33687]: cluster 2021-11-20T07:37: 2021-11-20T07:37:39.690 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:39 smithi071 conmon[33687]: 38.354263+0000 mgr.smithi071.ynqmwn (mgr.14162) 157 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:39 smithi112 conmon[36062]: cluster 2021-11-20T07:37:38.354263+0000 mgr.smithi071.ynqmwn (mgr.14162) 157 2021-11-20T07:37:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:39 smithi112 conmon[36062]: : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:40.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 -- 172.21.15.71:0/1581980027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f67c0 msgr2=0x7f79780f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:40.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 --2- 172.21.15.71:0/1581980027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f67c0 0x7f79780f6c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f796c004660 tx=0x7f796c009b30).stop 2021-11-20T07:37:40.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 -- 172.21.15.71:0/1581980027 shutdown_connections 2021-11-20T07:37:40.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 --2- 172.21.15.71:0/1581980027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f67c0 0x7f79780f6c20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:40.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 --2- 172.21.15.71:0/1581980027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f5520 0x7f79780f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:40.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 -- 172.21.15.71:0/1581980027 >> 172.21.15.71:0/1581980027 conn(0x7f79780f0ca0 msgr2=0x7f79780f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:40.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 -- 172.21.15.71:0/1581980027 shutdown_connections 2021-11-20T07:37:40.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 -- 172.21.15.71:0/1581980027 wait complete. 2021-11-20T07:37:40.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.145+0000 7f797c980700 1 Processor -- start 2021-11-20T07:37:40.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 -- start start 2021-11-20T07:37:40.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:40.154 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f67c0 0x7f79781063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:40.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79781069e0 con 0x7f79780f5520 2021-11-20T07:37:40.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7978107150 con 0x7f79780f67c0 2021-11-20T07:37:40.155 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:40.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47660/0 (socket says 172.21.15.71:47660) 2021-11-20T07:37:40.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 -- 172.21.15.71:0/1644068960 learned_addr learned my addr 172.21.15.71:0/1644068960 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:40.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797659c700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f67c0 0x7f79781063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:40.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 -- 172.21.15.71:0/1644068960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f67c0 msgr2=0x7f79781063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:40.156 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f67c0 0x7f79781063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:40.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 -- 172.21.15.71:0/1644068960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c005040 con 0x7f79780f5520 2021-11-20T07:37:40.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f7976d9d700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f79680093f0 tx=0x7f7968006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:40.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7968013640 con 0x7f79780f5520 2021-11-20T07:37:40.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7968013c50 con 0x7f79780f5520 2021-11-20T07:37:40.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978107430 con 0x7f79780f5520 2021-11-20T07:37:40.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7968012c80 con 0x7f79780f5520 2021-11-20T07:37:40.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.146+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781079f0 con 0x7f79780f5520 2021-11-20T07:37:40.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.147+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7978059100 con 0x7f79780f5520 2021-11-20T07:37:40.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.148+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f796801a060 con 0x7f79780f5520 2021-11-20T07:37:40.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.148+0000 7f795f7fe700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 0x7f796005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:40.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.148+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7968075b00 con 0x7f79780f5520 2021-11-20T07:37:40.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.148+0000 7f797659c700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 0x7f796005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:40.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.148+0000 7f797659c700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 0x7f796005e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f796c004630 tx=0x7f796c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:40.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.152+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7968048b80 con 0x7f79780f5520 2021-11-20T07:37:40.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.302+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f79780fddf0 con 0x7f79780f5520 2021-11-20T07:37:40.305 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.303+0000 7f795f7fe700 1 -- 172.21.15.71:0/1644068960 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f796804c1a0 con 0x7f79780f5520 2021-11-20T07:37:40.305 INFO:teuthology.orchestra.run.smithi071.stdout:38654705688 2021-11-20T07:37:40.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 msgr2=0x7f796005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:40.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 0x7f796005e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f796c004630 tx=0x7f796c009b30).stop 2021-11-20T07:37:40.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 msgr2=0x7f7978105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:40.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f79680093f0 tx=0x7f7968006ec0).stop 2021-11-20T07:37:40.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 shutdown_connections 2021-11-20T07:37:40.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f796005c2b0 0x7f796005e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:40.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.304+0000 7f797c980700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79780f5520 0x7f7978105eb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:40.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.305+0000 7f797c980700 1 --2- 172.21.15.71:0/1644068960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79780f67c0 0x7f79781063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:40.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.305+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 >> 172.21.15.71:0/1644068960 conn(0x7f79780f0ca0 msgr2=0x7f79780fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:40.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.305+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 shutdown_connections 2021-11-20T07:37:40.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:40.305+0000 7f797c980700 1 -- 172.21.15.71:0/1644068960 wait complete. 2021-11-20T07:37:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:40 smithi071 conmon[33687]: audit 2021-11-20T07:37:40.304432+0000 mon.smithi071 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.71:0/1644068960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T07:37:40.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705686 got 38654705688 for osd.0 2021-11-20T07:37:40.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-11-20T07:37:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:40 smithi112 conmon[36062]: audit 2021-11-20T07:37:40.304432+0000 mon.smithi071 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.71:0/1644068960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T07:37:41.104 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:41.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:41 smithi071 conmon[33687]: cluster 2021-11-20T07:37:40.354524+0000 mgr.smithi071.ynqmwn (mgr.14162) 158 2021-11-20T07:37:41.630 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:41 smithi071 conmon[33687]: : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:41 smithi112 conmon[36062]: cluster 2021-11-20T07:37:40.354524+0000 mgr.smithi071.ynqmwn (mgr.14162) 158 2021-11-20T07:37:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:41 smithi112 conmon[36062]: : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:42.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 -- 172.21.15.71:0/4277164471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 msgr2=0x7faf840f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:42.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 --2- 172.21.15.71:0/4277164471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840f5b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7faf80004660 tx=0x7faf80009b30).stop 2021-11-20T07:37:42.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 -- 172.21.15.71:0/4277164471 shutdown_connections 2021-11-20T07:37:42.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 --2- 172.21.15.71:0/4277164471 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf840f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 --2- 172.21.15.71:0/4277164471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840f5b00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 -- 172.21.15.71:0/4277164471 >> 172.21.15.71:0/4277164471 conn(0x7faf840f0e20 msgr2=0x7faf840f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:42.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 -- 172.21.15.71:0/4277164471 shutdown_connections 2021-11-20T07:37:42.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.686+0000 7faf8c14d700 1 -- 172.21.15.71:0/4277164471 wait complete. 2021-11-20T07:37:42.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 Processor -- start 2021-11-20T07:37:42.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 -- start start 2021-11-20T07:37:42.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:42.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf84061d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:42.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84062390 con 0x7faf840f56e0 2021-11-20T07:37:42.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf8c14d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf840624d0 con 0x7faf840f6980 2021-11-20T07:37:42.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:42.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47670/0 (socket says 172.21.15.71:47670) 2021-11-20T07:37:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 -- 172.21.15.71:0/2509726704 learned_addr learned my addr 172.21.15.71:0/2509726704 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf896e8700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf84061d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:42.702 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 -- 172.21.15.71:0/2509726704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 msgr2=0x7faf84061d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:42.702 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf84061d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.702 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf89ee9700 1 -- 172.21.15.71:0/2509726704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80005040 con 0x7faf840f56e0 2021-11-20T07:37:42.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.687+0000 7faf896e8700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf84061d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:37:42.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf89ee9700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7faf80004e30 tx=0x7faf80009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:42.704 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf8003a070 con 0x7faf840f56e0 2021-11-20T07:37:42.704 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf80013410 con 0x7faf840f56e0 2021-11-20T07:37:42.704 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf84062de0 con 0x7faf840f56e0 2021-11-20T07:37:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf8003e3c0 con 0x7faf840f56e0 2021-11-20T07:37:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.688+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf84063340 con 0x7faf840f56e0 2021-11-20T07:37:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.689+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84045ad0 con 0x7faf840f56e0 2021-11-20T07:37:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.689+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faf8003f420 con 0x7faf840f56e0 2021-11-20T07:37:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.689+0000 7faf7a7fc700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.690+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faf80099800 con 0x7faf840f56e0 2021-11-20T07:37:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.690+0000 7faf896e8700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.690+0000 7faf896e8700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 0x7faf7005e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7faf74009fd0 tx=0x7faf7400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:42.708 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.693+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf8006c800 con 0x7faf840f56e0 2021-11-20T07:37:42.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.843+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faf840fdf10 con 0x7faf840f56e0 2021-11-20T07:37:42.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.843+0000 7faf7a7fc700 1 -- 172.21.15.71:0/2509726704 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf8006fe20 con 0x7faf840f56e0 2021-11-20T07:37:42.846 INFO:teuthology.orchestra.run.smithi071.stdout:60129542165 2021-11-20T07:37:42.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 msgr2=0x7faf7005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:42.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 0x7faf7005e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7faf74009fd0 tx=0x7faf7400b040).stop 2021-11-20T07:37:42.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 msgr2=0x7faf840618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:42.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7faf80004e30 tx=0x7faf80009b30).stop 2021-11-20T07:37:42.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 shutdown_connections 2021-11-20T07:37:42.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7faf7005c2a0 0x7faf7005e760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf840f56e0 0x7faf840618a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 --2- 172.21.15.71:0/2509726704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faf840f6980 0x7faf84061d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:42.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 >> 172.21.15.71:0/2509726704 conn(0x7faf840f0e20 msgr2=0x7faf840fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:42.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 shutdown_connections 2021-11-20T07:37:42.851 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:42.845+0000 7faf8c14d700 1 -- 172.21.15.71:0/2509726704 wait complete. 2021-11-20T07:37:43.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542163 got 60129542165 for osd.1 2021-11-20T07:37:43.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-11-20T07:37:43.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:43 smithi071 conmon[33687]: cluster 2021-11-20T07:37:42.355123+0000 mgr.smithi071.ynqmwn (mgr.14162) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:43.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:43 smithi071 conmon[33687]: audit 2021-11-20T07:37:42.844812+0000 mon.smithi071 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.71:0/2509726704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T07:37:43.755 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:43 smithi112 conmon[36062]: cluster 2021-11-20T07:37:42.355123+0000 mgr.smithi071.ynqmwn (mgr.14162) 159 : cluster 2021-11-20T07:37:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:43 smithi112 conmon[36062]: [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:43 smithi112 conmon[36062]: audit 2021-11-20T07:37:42.844812+0000 mon.smithi071 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.71:0/2509726704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T07:37:44.935 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 -- 172.21.15.71:0/196858031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 msgr2=0x7fb4240f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:44.935 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 --2- 172.21.15.71:0/196858031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb4240f6750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb410004660 tx=0x7fb410009b30).stop 2021-11-20T07:37:44.935 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 -- 172.21.15.71:0/196858031 shutdown_connections 2021-11-20T07:37:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 --2- 172.21.15.71:0/196858031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb4240f6750 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 --2- 172.21.15.71:0/196858031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4240f7ce0 0x7fb4240f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.932+0000 7fb429dec700 1 -- 172.21.15.71:0/196858031 >> 172.21.15.71:0/196858031 conn(0x7fb4240f1d70 msgr2=0x7fb4240f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 -- 172.21.15.71:0/196858031 shutdown_connections 2021-11-20T07:37:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 -- 172.21.15.71:0/196858031 wait complete. 2021-11-20T07:37:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 Processor -- start 2021-11-20T07:37:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 -- start start 2021-11-20T07:37:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4240f7ce0 0x7fb424069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.933+0000 7fb429dec700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406a380 con 0x7fb4240f62f0 2021-11-20T07:37:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb429dec700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406a4c0 con 0x7fb4240f7ce0 2021-11-20T07:37:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47680/0 (socket says 172.21.15.71:47680) 2021-11-20T07:37:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 -- 172.21.15.71:0/2448103324 learned_addr learned my addr 172.21.15.71:0/2448103324 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 -- 172.21.15.71:0/2448103324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4240f7ce0 msgr2=0x7fb424069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4240f7ce0 0x7fb424069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 -- 172.21.15.71:0/2448103324 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb410005040 con 0x7fb4240f62f0 2021-11-20T07:37:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb428dea700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb41800c790 tx=0x7fb418007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41800e680 con 0x7fb4240f62f0 2021-11-20T07:37:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42406a7a0 con 0x7fb4240f62f0 2021-11-20T07:37:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb41800ec90 con 0x7fb4240f62f0 2021-11-20T07:37:44.941 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb418012cb0 con 0x7fb4240f62f0 2021-11-20T07:37:44.941 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.934+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42410efd0 con 0x7fb4240f62f0 2021-11-20T07:37:44.942 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.935+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb424045ad0 con 0x7fb4240f62f0 2021-11-20T07:37:44.943 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.935+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb418010070 con 0x7fb4240f62f0 2021-11-20T07:37:44.943 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.936+0000 7fb4217fa700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 0x7fb41405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:44.943 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.936+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb418075c90 con 0x7fb4240f62f0 2021-11-20T07:37:44.943 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.936+0000 7fb423fff700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 0x7fb41405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:44.944 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.936+0000 7fb423fff700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 0x7fb41405e770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb410009f50 tx=0x7fb410009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:44.944 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:44.940+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb418048c90 con 0x7fb4240f62f0 2021-11-20T07:37:45.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.087+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb4240feea0 con 0x7fb4240f62f0 2021-11-20T07:37:45.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.088+0000 7fb4217fa700 1 -- 172.21.15.71:0/2448103324 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb41804c2b0 con 0x7fb4240f62f0 2021-11-20T07:37:45.090 INFO:teuthology.orchestra.run.smithi071.stdout:77309411347 2021-11-20T07:37:45.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 msgr2=0x7fb41405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:45.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 0x7fb41405e770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb410009f50 tx=0x7fb410009b30).stop 2021-11-20T07:37:45.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 msgr2=0x7fb42406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:45.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb41800c790 tx=0x7fb418007d90).stop 2021-11-20T07:37:45.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 shutdown_connections 2021-11-20T07:37:45.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb41405c2b0 0x7fb41405e770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:45.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4240f62f0 0x7fb42406b9a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:45.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 --2- 172.21.15.71:0/2448103324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4240f7ce0 0x7fb424069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:45.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.089+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 >> 172.21.15.71:0/2448103324 conn(0x7fb4240f1d70 msgr2=0x7fb4240fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:45.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.090+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 shutdown_connections 2021-11-20T07:37:45.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:45.090+0000 7fb429dec700 1 -- 172.21.15.71:0/2448103324 wait complete. 2021-11-20T07:37:45.466 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411345 got 77309411347 for osd.2 2021-11-20T07:37:45.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-11-20T07:37:45.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:45 smithi071 conmon[33687]: cluster 2021-11-20T07:37:44.355549+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:37:45.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:45 smithi071 conmon[33687]: 14162) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:45.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:45 smithi071 conmon[33687]: audit 2021-11-20T07:37:45.088377+0000 mon.smithi071 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.71:0/2448103324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T07:37:45.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:45 smithi112 conmon[36062]: cluster 2021-11-20T07:37:44.355549+0000 mgr.smithi071.ynqmwn (mgr.14162) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:45.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:45 smithi112 conmon[36062]: audit 2021-11-20T07:37:45.088377+0000 mon.smithi071 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.71:0/2448103324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T07:37:45.900 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:47.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.070+0000 7f2612e23700 1 -- 172.21.15.71:0/1110382889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f56e0 msgr2=0x7f260c0f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:47.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.070+0000 7f2612e23700 1 --2- 172.21.15.71:0/1110382889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f56e0 0x7f260c0f5b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f26080099e0 tx=0x7f2608004d10).stop 2021-11-20T07:37:47.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.070+0000 7f2612e23700 1 -- 172.21.15.71:0/1110382889 shutdown_connections 2021-11-20T07:37:47.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.070+0000 7f2612e23700 1 --2- 172.21.15.71:0/1110382889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f6980 0x7f260c0f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 --2- 172.21.15.71:0/1110382889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f56e0 0x7f260c0f5b00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- 172.21.15.71:0/1110382889 >> 172.21.15.71:0/1110382889 conn(0x7f260c0f0e20 msgr2=0x7f260c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:47.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- 172.21.15.71:0/1110382889 shutdown_connections 2021-11-20T07:37:47.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- 172.21.15.71:0/1110382889 wait complete. 2021-11-20T07:37:47.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 Processor -- start 2021-11-20T07:37:47.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- start start 2021-11-20T07:37:47.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f56e0 0x7f260c1060f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:47.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:47.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c106b90 con 0x7f260c0f6980 2021-11-20T07:37:47.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.071+0000 7f2612e23700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c106cd0 con 0x7f260c0f56e0 2021-11-20T07:37:47.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:47.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47692/0 (socket says 172.21.15.71:47692) 2021-11-20T07:37:47.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 -- 172.21.15.71:0/2295799360 learned_addr learned my addr 172.21.15.71:0/2295799360 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:47.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2610bbf700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f56e0 0x7f260c1060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:47.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 -- 172.21.15.71:0/2295799360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f56e0 msgr2=0x7f260c1060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:47.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f56e0 0x7f260c1060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 -- 172.21.15.71:0/2295799360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608009640 con 0x7f260c0f6980 2021-11-20T07:37:47.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2603fff700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f25fc00c530 tx=0x7f25fc007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:47.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25fc00f650 con 0x7f260c0f6980 2021-11-20T07:37:47.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c107670 con 0x7f260c0f6980 2021-11-20T07:37:47.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25fc00fc60 con 0x7f260c0f6980 2021-11-20T07:37:47.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25fc012c80 con 0x7f260c0f6980 2021-11-20T07:37:47.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.072+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c107bb0 con 0x7f260c0f6980 2021-11-20T07:37:47.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.073+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c045ad0 con 0x7f260c0f6980 2021-11-20T07:37:47.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.073+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f25fc010070 con 0x7f260c0f6980 2021-11-20T07:37:47.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.074+0000 7f26017fa700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 0x7f25f4060970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:47.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.074+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f25fc075800 con 0x7f260c0f6980 2021-11-20T07:37:47.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.074+0000 7f2610bbf700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 0x7f25f4060970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:47.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.074+0000 7f2610bbf700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 0x7f25f4060970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f2608000f80 tx=0x7f2608004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:47.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.078+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25fc0452b0 con 0x7f260c0f6980 2021-11-20T07:37:47.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.224+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f260c0ff8d0 con 0x7f260c0f6980 2021-11-20T07:37:47.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.225+0000 7f26017fa700 1 -- 172.21.15.71:0/2295799360 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f25fc048880 con 0x7f260c0f6980 2021-11-20T07:37:47.226 INFO:teuthology.orchestra.run.smithi071.stdout:107374182416 2021-11-20T07:37:47.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 msgr2=0x7f25f4060970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:47.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 0x7f25f4060970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f2608000f80 tx=0x7f2608004d10).stop 2021-11-20T07:37:47.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 msgr2=0x7f260c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:47.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f25fc00c530 tx=0x7f25fc007e20).stop 2021-11-20T07:37:47.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 shutdown_connections 2021-11-20T07:37:47.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f260c0f56e0 0x7f260c1060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f25f405e4b0 0x7f25f4060970 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 --2- 172.21.15.71:0/2295799360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f260c0f6980 0x7f260c1065e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:47.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.226+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 >> 172.21.15.71:0/2295799360 conn(0x7f260c0f0e20 msgr2=0x7f260c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:47.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.227+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 shutdown_connections 2021-11-20T07:37:47.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:47.227+0000 7f2612e23700 1 -- 172.21.15.71:0/2295799360 wait complete. 2021-11-20T07:37:47.511 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182413 got 107374182416 for osd.3 2021-11-20T07:37:47.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-11-20T07:37:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:47 smithi112 conmon[36062]: cluster 2021-11-20T07:37:46.355850+0000 mgr.smithi071.ynqmwn (mgr.14162) 161 : cluster 2021-11-20T07:37:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:47 smithi112 conmon[36062]: [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:47.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:47 smithi112 conmon[36062]: audit 2021-11-20T07:37:47.225370+0000 mon.smithi071 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.71:0/2295799360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T07:37:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:47 smithi071 conmon[33687]: cluster 2021-11-20T07:37:46.355850+0000 mgr.smithi071.ynqmwn (mgr.14162) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:47 smithi071 conmon[33687]: audit 2021-11-20T07:37:47.225370+0000 mon.smithi071 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.71:0/2295799360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T07:37:47.945 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:49.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- 172.21.15.71:0/4223746223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f7a60 msgr2=0x7f0c900f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:49.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 --2- 172.21.15.71:0/4223746223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f7a60 0x7f0c900f7ec0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0c80004660 tx=0x7f0c80009b30).stop 2021-11-20T07:37:49.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- 172.21.15.71:0/4223746223 shutdown_connections 2021-11-20T07:37:49.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 --2- 172.21.15.71:0/4223746223 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f7a60 0x7f0c900f7ec0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:49.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 --2- 172.21.15.71:0/4223746223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f67c0 0x7f0c900f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:49.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- 172.21.15.71:0/4223746223 >> 172.21.15.71:0/4223746223 conn(0x7f0c900f1f20 msgr2=0x7f0c900f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:49.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- 172.21.15.71:0/4223746223 shutdown_connections 2021-11-20T07:37:49.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- 172.21.15.71:0/4223746223 wait complete. 2021-11-20T07:37:49.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 Processor -- start 2021-11-20T07:37:49.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.104+0000 7f0c98495700 1 -- start start 2021-11-20T07:37:49.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:49.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f7a60 0x7f0c90069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:49.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9006a410 con 0x7f0c900f67c0 2021-11-20T07:37:49.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9006a550 con 0x7f0c900f7a60 2021-11-20T07:37:49.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:49.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47700/0 (socket says 172.21.15.71:47700) 2021-11-20T07:37:49.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 -- 172.21.15.71:0/1406874712 learned_addr learned my addr 172.21.15.71:0/1406874712 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:49.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c95a30700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f7a60 0x7f0c90069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:49.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 -- 172.21.15.71:0/1406874712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f7a60 msgr2=0x7f0c90069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:49.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f7a60 0x7f0c90069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:49.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 -- 172.21.15.71:0/1406874712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80005040 con 0x7f0c900f67c0 2021-11-20T07:37:49.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c96231700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0c8c0093f0 tx=0x7f0c8c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:49.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c8c013640 con 0x7f0c900f67c0 2021-11-20T07:37:49.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c8c013c50 con 0x7f0c900f67c0 2021-11-20T07:37:49.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9006a830 con 0x7f0c900f67c0 2021-11-20T07:37:49.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c8c012c80 con 0x7f0c900f67c0 2021-11-20T07:37:49.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.105+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9010f0e0 con 0x7f0c900f67c0 2021-11-20T07:37:49.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.106+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c90045ad0 con 0x7f0c900f67c0 2021-11-20T07:37:49.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.107+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0c8c01a060 con 0x7f0c900f67c0 2021-11-20T07:37:49.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.107+0000 7f0c86ffd700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 0x7f0c7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:49.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.107+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0c8c075b90 con 0x7f0c900f67c0 2021-11-20T07:37:49.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.107+0000 7f0c95a30700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 0x7f0c7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:49.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.108+0000 7f0c95a30700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 0x7f0c7c05e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0c80004630 tx=0x7f0c80009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:49.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.111+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c8c048b90 con 0x7f0c900f67c0 2021-11-20T07:37:49.260 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.259+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0c90027a00 con 0x7f0c900f67c0 2021-11-20T07:37:49.260 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.260+0000 7f0c86ffd700 1 -- 172.21.15.71:0/1406874712 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0c8c04c1b0 con 0x7f0c900f67c0 2021-11-20T07:37:49.261 INFO:teuthology.orchestra.run.smithi071.stdout:133143986188 2021-11-20T07:37:49.261 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 msgr2=0x7f0c7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:49.262 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 0x7f0c7c05e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0c80004630 tx=0x7f0c80009b30).stop 2021-11-20T07:37:49.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 msgr2=0x7f0c9006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:49.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0c8c0093f0 tx=0x7f0c8c006ec0).stop 2021-11-20T07:37:49.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 shutdown_connections 2021-11-20T07:37:49.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0c7c05c2a0 0x7f0c7c05e760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:49.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c900f67c0 0x7f0c9006b9a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:49.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 --2- 172.21.15.71:0/1406874712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0c900f7a60 0x7f0c90069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:49.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 >> 172.21.15.71:0/1406874712 conn(0x7f0c900f1f20 msgr2=0x7f0c900fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:49.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 shutdown_connections 2021-11-20T07:37:49.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:49.261+0000 7f0c98495700 1 -- 172.21.15.71:0/1406874712 wait complete. 2021-11-20T07:37:49.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:49 smithi071 conmon[33687]: cluster 2021-11-20T07:37:48.356183+0000 mgr.smithi071.ynqmwn (mgr.14162) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:49.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:49 smithi071 conmon[33687]: audit 2021-11-20T07:37:49.260149+0000 mon.smithi071 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.71:0/1406874712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T07:37:49.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:49 smithi112 conmon[36062]: cluster 2021-11-20T07:37:48.356183+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:37:49.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:49 smithi112 conmon[36062]: 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:49.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:49 smithi112 conmon[36062]: audit 2021-11-20T07:37:49.260149+0000 mon.smithi071 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.71:0/1406874712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T07:37:49.845 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986186 got 133143986188 for osd.4 2021-11-20T07:37:49.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-11-20T07:37:50.280 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:51.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 -- 172.21.15.71:0/690350267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f5520 msgr2=0x7f19a80f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:51.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 --2- 172.21.15.71:0/690350267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f5520 0x7f19a80f5940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f199c004660 tx=0x7f199c009b30).stop 2021-11-20T07:37:51.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 -- 172.21.15.71:0/690350267 shutdown_connections 2021-11-20T07:37:51.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 --2- 172.21.15.71:0/690350267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f67c0 0x7f19a80f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:51.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 --2- 172.21.15.71:0/690350267 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f5520 0x7f19a80f5940 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:51.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 -- 172.21.15.71:0/690350267 >> 172.21.15.71:0/690350267 conn(0x7f19a80f0ca0 msgr2=0x7f19a80f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:51.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 -- 172.21.15.71:0/690350267 shutdown_connections 2021-11-20T07:37:51.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.462+0000 7f19adf1f700 1 -- 172.21.15.71:0/690350267 wait complete. 2021-11-20T07:37:51.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 Processor -- start 2021-11-20T07:37:51.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 -- start start 2021-11-20T07:37:51.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f5520 0x7f19a8105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:51.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:51.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a81069f0 con 0x7f19a80f67c0 2021-11-20T07:37:51.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19adf1f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8107160 con 0x7f19a80f5520 2021-11-20T07:37:51.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:51.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47712/0 (socket says 172.21.15.71:47712) 2021-11-20T07:37:51.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19a7fff700 1 -- 172.21.15.71:0/1439607168 learned_addr learned my addr 172.21.15.71:0/1439607168 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:51.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.463+0000 7f19acf1d700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f5520 0x7f19a8105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:51.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a7fff700 1 -- 172.21.15.71:0/1439607168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f5520 msgr2=0x7f19a8105ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:51.480 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a7fff700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f5520 0x7f19a8105ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:51.481 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a7fff700 1 -- 172.21.15.71:0/1439607168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c005040 con 0x7f19a80f67c0 2021-11-20T07:37:51.481 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a7fff700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f199400b410 tx=0x7f1994006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:51.482 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19940125f0 con 0x7f19a80f67c0 2021-11-20T07:37:51.482 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f199400be30 con 0x7f19a80f67c0 2021-11-20T07:37:51.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a8107440 con 0x7f19a80f67c0 2021-11-20T07:37:51.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1994002a20 con 0x7f19a80f67c0 2021-11-20T07:37:51.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.464+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a8107a00 con 0x7f19a80f67c0 2021-11-20T07:37:51.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.465+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a8059100 con 0x7f19a80f67c0 2021-11-20T07:37:51.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.465+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f199400f030 con 0x7f19a80f67c0 2021-11-20T07:37:51.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.466+0000 7f19a57fa700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 0x7f199805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:51.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.466+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1994075f80 con 0x7f19a80f67c0 2021-11-20T07:37:51.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.466+0000 7f19acf1d700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 0x7f199805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:51.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.466+0000 7f19acf1d700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 0x7f199805e770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f199c004630 tx=0x7f199c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:51.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.470+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1994048f80 con 0x7f19a80f67c0 2021-11-20T07:37:51.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:51 smithi071 conmon[33687]: cluster 2021-11-20T07:37:50.356705+0000 mgr.smithi071.ynqmwn (mgr.14162) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:51.621 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.620+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f19a80fddd0 con 0x7f19a80f67c0 2021-11-20T07:37:51.621 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.620+0000 7f19a57fa700 1 -- 172.21.15.71:0/1439607168 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f199404c5a0 con 0x7f19a80f67c0 2021-11-20T07:37:51.622 INFO:teuthology.orchestra.run.smithi071.stdout:150323855369 2021-11-20T07:37:51.623 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 msgr2=0x7f199805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:51.623 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 0x7f199805e770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f199c004630 tx=0x7f199c009b30).stop 2021-11-20T07:37:51.623 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 msgr2=0x7f19a81063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:51.624 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f199400b410 tx=0x7f1994006ef0).stop 2021-11-20T07:37:51.624 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 shutdown_connections 2021-11-20T07:37:51.624 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19a80f5520 0x7f19a8105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:51.624 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f199805c2b0 0x7f199805e770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:51.625 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 --2- 172.21.15.71:0/1439607168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19a80f67c0 0x7f19a81063b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:51.625 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 >> 172.21.15.71:0/1439607168 conn(0x7f19a80f0ca0 msgr2=0x7f19a80fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:51.625 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 shutdown_connections 2021-11-20T07:37:51.625 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:51.622+0000 7f19adf1f700 1 -- 172.21.15.71:0/1439607168 wait complete. 2021-11-20T07:37:51.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:51 smithi112 conmon[36062]: cluster 2021-11-20T07:37:50.356705+0000 mgr.smithi071.ynqmwn (mgr.14162) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:51.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:51 smithi112 conmon[36062]: 2021-11-20T07:37:51.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855367 got 150323855369 for osd.5 2021-11-20T07:37:51.990 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-11-20T07:37:51.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T07:37:52.427 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:52.623 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:52 smithi071 conmon[33687]: audit 2021-11-20T07:37:51.620909+0000 mon.smithi071 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.71:0/1439607168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T07:37:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:52 smithi112 conmon[36062]: audit 2021-11-20T07:37:51.620909+0000 mon.smithi071 (mon. 2021-11-20T07:37:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:52 smithi112 conmon[36062]: 0) 437 : audit [DBG] from='client.? 172.21.15.71:0/1439607168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T07:37:53.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- 172.21.15.71:0/984962920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f56e0 msgr2=0x7fefc40f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:53.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 --2- 172.21.15.71:0/984962920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f56e0 0x7fefc40f5b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fefb8004660 tx=0x7fefb8009b30).stop 2021-11-20T07:37:53.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- 172.21.15.71:0/984962920 shutdown_connections 2021-11-20T07:37:53.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 --2- 172.21.15.71:0/984962920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f6980 0x7fefc40f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 --2- 172.21.15.71:0/984962920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f56e0 0x7fefc40f5b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- 172.21.15.71:0/984962920 >> 172.21.15.71:0/984962920 conn(0x7fefc40f0e20 msgr2=0x7fefc40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:53.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- 172.21.15.71:0/984962920 shutdown_connections 2021-11-20T07:37:53.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- 172.21.15.71:0/984962920 wait complete. 2021-11-20T07:37:53.698 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:53 smithi071 conmon[33687]: cluster 2021-11-20T07:37:52.357005+0000 mgr.smithi071.ynqmwn (mgr.14162) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:53.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 Processor -- start 2021-11-20T07:37:53.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.693+0000 7fefcae67700 1 -- start start 2021-11-20T07:37:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f56e0 0x7fefc4109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:53.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4060940 con 0x7fefc40f6980 2021-11-20T07:37:53.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4060a80 con 0x7fefc40f56e0 2021-11-20T07:37:53.705 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:53.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47722/0 (socket says 172.21.15.71:47722) 2021-11-20T07:37:53.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 -- 172.21.15.71:0/4162437982 learned_addr learned my addr 172.21.15.71:0/4162437982 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:53.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc8c03700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f56e0 0x7fefc4109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:53.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 -- 172.21.15.71:0/4162437982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f56e0 msgr2=0x7fefc4109d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:53.708 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f56e0 0x7fefc4109d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.708 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 -- 172.21.15.71:0/4162437982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8005040 con 0x7fefc40f6980 2021-11-20T07:37:53.709 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc3fff700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fefb000e7f0 tx=0x7fefb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:53.709 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefb00092e0 con 0x7fefc40f6980 2021-11-20T07:37:53.709 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc410c830 con 0x7fefc40f6980 2021-11-20T07:37:53.710 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefb000f040 con 0x7fefc40f6980 2021-11-20T07:37:53.710 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc410cdf0 con 0x7fefc40f6980 2021-11-20T07:37:53.711 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.694+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefb0014750 con 0x7fefc40f6980 2021-11-20T07:37:53.711 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fefb001b030 con 0x7fefc40f6980 2021-11-20T07:37:53.712 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc4045ad0 con 0x7fefc40f6980 2021-11-20T07:37:53.712 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefc17fa700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 0x7fefb405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:53.712 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fefb0075950 con 0x7fefc40f6980 2021-11-20T07:37:53.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefc8c03700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 0x7fefb405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:53.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.696+0000 7fefc8c03700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 0x7fefb405e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fefb8004630 tx=0x7fefb8016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:53.713 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.700+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefb0048a00 con 0x7fefc40f6980 2021-11-20T07:37:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:53 smithi112 conmon[36062]: cluster 2021-11-20T07:37:52.357005+0000 mgr.smithi071.ynqmwn 2021-11-20T07:37:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:53 smithi112 conmon[36062]: (mgr.14162) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:53.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.845+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fefc40fdfb0 con 0x7fefb405c2a0 2021-11-20T07:37:53.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.846+0000 7fefc17fa700 1 -- 172.21.15.71:0/4162437982 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24098 (secure 0 0 0) 0x7fefc40fdfb0 con 0x7fefb405c2a0 2021-11-20T07:37:53.848 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:53.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.848+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 msgr2=0x7fefb405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:53.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.848+0000 7fefcae67700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 0x7fefb405e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fefb8004630 tx=0x7fefb8016040).stop 2021-11-20T07:37:53.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.848+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 msgr2=0x7fefc410c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:53.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.848+0000 7fefcae67700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fefb000e7f0 tx=0x7fefb000b040).stop 2021-11-20T07:37:53.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 shutdown_connections 2021-11-20T07:37:53.851 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fefc40f56e0 0x7fefc4109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.851 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fefb405c2a0 0x7fefb405e760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.851 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 --2- 172.21.15.71:0/4162437982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40f6980 0x7fefc410c280 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:53.851 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 >> 172.21.15.71:0/4162437982 conn(0x7fefc40f0e20 msgr2=0x7fefc40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:53.852 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 shutdown_connections 2021-11-20T07:37:53.852 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:53.849+0000 7fefcae67700 1 -- 172.21.15.71:0/4162437982 wait complete. 2021-11-20T07:37:53.852 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2021-11-20T07:37:54.237 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2021-11-20T07:37:52.356886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31000,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":29056,"kb_avail":562439912,"statfs":{"total":575970213888,"available":575938469888,"internally_reserved":0,"allocated":1941504,"data_stored":725196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29753344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"33","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-20T07:37:15.907711+0000","last_change":"2021-11-20T07:36:58.617600+0000","last_active":"2021-11-20T07:37:15.907711+0000","last_peered":"2021-11-20T07:37:15.907711+0000","last_clean":"2021-11-20T07:37:15.907711+0000","last_became_active":"2021-11-20T07:36:58.617263+0000","last_became_peered":"2021-11-20T07:36:58.617263+0000","last_unstale":"2021-11-20T07:37:15.907711+0000","last_undegraded":"2021-11-20T07:37:15.907711+0000","last_fullsized":"2021-11-20T07:37:15.907711+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T07:36:36.056365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T07:36:36.056365+0000","last_clean_scrub_stamp":"2021-11-20T07:36:36.056365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:25 2021","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.28100000000000003}]},{"osd":1,"last update":"Sat Nov 20 07:37:25 2021","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.255},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31}]},{"osd":3,"last update":"Sat Nov 20 07:37:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.441},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 07:37:12 2021","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Nov 20 07:37:27 2021","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:10 2021","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.252}]},{"osd":2,"last update":"Sat Nov 20 07:37:26 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.313},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Nov 20 07:37:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T07:37:54.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T07:37:54.674 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:55.669 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:55 smithi071 conmon[33687]: audit 2021-11-20T07:37:53.846104+0000 mgr.smithi071.ynqmwn (mgr.14162) 165 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T07:37:55.669 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:55 smithi071 conmon[33687]: cluster 2021-11-20T07:37:54.357303+0000 mgr.smithi071.ynqmwn (mgr.14162) 166 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:55 smithi112 conmon[36062]: audit 2021-11-20T07:37:53.846104+0000 mgr.smithi071.ynqmwn (mgr.14162) 165 2021-11-20T07:37:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:55 smithi112 conmon[36062]: : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T07:37:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:55 smithi112 conmon[36062]: cluster 2021-11-20T07:37:54.357303+0000 mgr.smithi071.ynqmwn (mgr.14162) 166 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 -- 172.21.15.71:0/24632191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 msgr2=0x7f20140f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 --2- 172.21.15.71:0/24632191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f7bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2008004660 tx=0x7f2008009b30).stop 2021-11-20T07:37:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 -- 172.21.15.71:0/24632191 shutdown_connections 2021-11-20T07:37:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 --2- 172.21.15.71:0/24632191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f7bc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 --2- 172.21.15.71:0/24632191 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 0x7f20140f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 -- 172.21.15.71:0/24632191 >> 172.21.15.71:0/24632191 conn(0x7f20140f05e0 msgr2=0x7f20140f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 -- 172.21.15.71:0/24632191 shutdown_connections 2021-11-20T07:37:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 -- 172.21.15.71:0/24632191 wait complete. 2021-11-20T07:37:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.953+0000 7f201a19d700 1 Processor -- start 2021-11-20T07:37:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 -- start start 2021-11-20T07:37:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 0x7f20140f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20140f6030 con 0x7f20140f5440 2021-11-20T07:37:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20140f6170 con 0x7f20140f4aa0 2021-11-20T07:37:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47732/0 (socket says 172.21.15.71:47732) 2021-11-20T07:37:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 -- 172.21.15.71:0/1705221008 learned_addr learned my addr 172.21.15.71:0/1705221008 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f20137fe700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 0x7f20140f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 -- 172.21.15.71:0/1705221008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 msgr2=0x7f20140f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 0x7f20140f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 -- 172.21.15.71:0/1705221008 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008005040 con 0x7f20140f5440 2021-11-20T07:37:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f2012ffd700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2008004e30 tx=0x7f2008009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2008018070 con 0x7f20140f5440 2021-11-20T07:37:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.954+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20140f63f0 con 0x7f20140f5440 2021-11-20T07:37:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.955+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2008013410 con 0x7f20140f5440 2021-11-20T07:37:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.955+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20140f6950 con 0x7f20140f5440 2021-11-20T07:37:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.955+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f200801c3c0 con 0x7f20140f5440 2021-11-20T07:37:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.955+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2014045ad0 con 0x7f20140f5440 2021-11-20T07:37:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.956+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f200801d420 con 0x7f20140f5440 2021-11-20T07:37:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.956+0000 7f1ffbfff700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 0x7f1ffc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.956+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f20080777d0 con 0x7f20140f5440 2021-11-20T07:37:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.956+0000 7f20137fe700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 0x7f1ffc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.957+0000 7f20137fe700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 0x7f1ffc05e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f20140f73e0 tx=0x7f2004004d00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:55.960+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f200804a7d0 con 0x7f20140f5440 2021-11-20T07:37:56.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.106+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20140ff0a0 con 0x7f1ffc05c2a0 2021-11-20T07:37:56.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.107+0000 7f1ffbfff700 1 -- 172.21.15.71:0/1705221008 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24126 (secure 0 0 0) 0x7f20140ff0a0 con 0x7f1ffc05c2a0 2021-11-20T07:37:56.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:56.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.108+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 msgr2=0x7f1ffc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:56.110 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.108+0000 7f201a19d700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 0x7f1ffc05e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f20140f73e0 tx=0x7f2004004d00).stop 2021-11-20T07:37:56.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.108+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 msgr2=0x7f20140f5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:56.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.108+0000 7f201a19d700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2008004e30 tx=0x7f2008009b30).stop 2021-11-20T07:37:56.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 shutdown_connections 2021-11-20T07:37:56.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20140f4aa0 0x7f20140f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:56.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1ffc05c2a0 0x7f1ffc05e760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:56.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 --2- 172.21.15.71:0/1705221008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20140f5440 0x7f20140f5ad0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:56.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 >> 172.21.15.71:0/1705221008 conn(0x7f20140f05e0 msgr2=0x7f20140fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:56.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 shutdown_connections 2021-11-20T07:37:56.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:56.109+0000 7f201a19d700 1 -- 172.21.15.71:0/1705221008 wait complete. 2021-11-20T07:37:56.114 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2021-11-20T07:37:56.487 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2021-11-20T07:37:54.357174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31000,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":29056,"kb_avail":562439912,"statfs":{"total":575970213888,"available":575938469888,"internally_reserved":0,"allocated":1941504,"data_stored":725196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29753344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"33","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-20T07:37:15.907711+0000","last_change":"2021-11-20T07:36:58.617600+0000","last_active":"2021-11-20T07:37:15.907711+0000","last_peered":"2021-11-20T07:37:15.907711+0000","last_clean":"2021-11-20T07:37:15.907711+0000","last_became_active":"2021-11-20T07:36:58.617263+0000","last_became_peered":"2021-11-20T07:36:58.617263+0000","last_unstale":"2021-11-20T07:37:15.907711+0000","last_undegraded":"2021-11-20T07:37:15.907711+0000","last_fullsized":"2021-11-20T07:37:15.907711+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T07:36:36.056365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T07:36:36.056365+0000","last_clean_scrub_stamp":"2021-11-20T07:36:36.056365+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.502},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:25 2021","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.28100000000000003}]},{"osd":1,"last update":"Sat Nov 20 07:37:25 2021","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.255},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31}]},{"osd":3,"last update":"Sat Nov 20 07:37:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.441},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 07:37:12 2021","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Nov 20 07:37:27 2021","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Nov 20 07:37:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 07:37:10 2021","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.253},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Nov 20 07:37:26 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.311},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Nov 20 07:37:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T07:37:56.488 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-11-20T07:37:56.489 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-11-20T07:37:56.489 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-11-20T07:37:56.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- ceph health --format=json 2021-11-20T07:37:56.924 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/mon.smithi071/config 2021-11-20T07:37:57.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:57 smithi071 conmon[33687]: audit 2021-11-20T07:37:56.107517+0000 mgr.smithi071.ynqmwn (mgr.14162) 167 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T07:37:57.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:57 smithi071 conmon[33687]: cluster 2021-11-20T07:37:56.357755+0000 mgr.smithi071.ynqmwn (mgr.14162) 168 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:57 smithi112 conmon[36062]: audit 2021-11-20T07:37:56.107517+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:37:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:57 smithi112 conmon[36062]: ) 167 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T07:37:57.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:57 smithi112 conmon[36062]: cluster 2021-11-20T07:37:56.357755+0000 mgr.smithi071.ynqmwn (mgr.14162) 168 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:58.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 -- 172.21.15.71:0/3372820580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 msgr2=0x7f91900f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:58.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 --2- 172.21.15.71:0/3372820580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f91900f5b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f918c004660 tx=0x7f918c009b30).stop 2021-11-20T07:37:58.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 -- 172.21.15.71:0/3372820580 shutdown_connections 2021-11-20T07:37:58.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 --2- 172.21.15.71:0/3372820580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f91900f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:37:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 --2- 172.21.15.71:0/3372820580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f91900f5b00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 -- 172.21.15.71:0/3372820580 >> 172.21.15.71:0/3372820580 conn(0x7f91900f0e20 msgr2=0x7f91900f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 -- 172.21.15.71:0/3372820580 shutdown_connections 2021-11-20T07:37:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.057+0000 7f91986d7700 1 -- 172.21.15.71:0/3372820580 wait complete. 2021-11-20T07:37:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 Processor -- start 2021-11-20T07:37:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 -- start start 2021-11-20T07:37:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f919010a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190060a10 con 0x7f91900f56e0 2021-11-20T07:37:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f91986d7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190060b50 con 0x7f91900f6980 2021-11-20T07:37:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47738/0 (socket says 172.21.15.71:47738) 2021-11-20T07:37:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 -- 172.21.15.71:0/1600477268 learned_addr learned my addr 172.21.15.71:0/1600477268 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:37:58.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9195c72700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f919010a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:58.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 -- 172.21.15.71:0/1600477268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 msgr2=0x7f919010a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:58.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f919010a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:58.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 -- 172.21.15.71:0/1600477268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c005040 con 0x7f91900f56e0 2021-11-20T07:37:58.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9195c72700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f919010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:37:58.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9196473700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f918c009f50 tx=0x7f918c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:58.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c018070 con 0x7f91900f56e0 2021-11-20T07:37:58.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.058+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f918c013410 con 0x7f91900f56e0 2021-11-20T07:37:58.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.059+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919010a6b0 con 0x7f91900f56e0 2021-11-20T07:37:58.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.059+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918c01c620 con 0x7f91900f56e0 2021-11-20T07:37:58.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.059+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919010ac10 con 0x7f91900f56e0 2021-11-20T07:37:58.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919010bfb0 con 0x7f91900f56e0 2021-11-20T07:37:58.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f918c01d420 con 0x7f91900f56e0 2021-11-20T07:37:58.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f9186ffd700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 0x7f917c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:37:58.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f918c0779f0 con 0x7f91900f56e0 2021-11-20T07:37:58.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f9195c72700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 0x7f917c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:37:58.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.060+0000 7f9195c72700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 0x7f917c05e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f918000a7e0 tx=0x7f9180008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:37:58.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.064+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f918c079e60 con 0x7f91900f56e0 2021-11-20T07:37:58.275 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.273+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91900fdfa0 con 0x7f91900f56e0 2021-11-20T07:37:58.275 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.274+0000 7f9186ffd700 1 -- 172.21.15.71:0/1600477268 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f918c04a9f0 con 0x7f91900f56e0 2021-11-20T07:37:58.276 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:37:58.277 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 msgr2=0x7f917c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:58.277 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 0x7f917c05e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f918000a7e0 tx=0x7f9180008040).stop 2021-11-20T07:37:58.278 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 msgr2=0x7f9190107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:37:58.278 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f918c009f50 tx=0x7f918c009b30).stop 2021-11-20T07:37:58.278 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 shutdown_connections 2021-11-20T07:37:58.278 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.275+0000 7f91986d7700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f917c05c2a0 0x7f917c05e760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:58.279 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.276+0000 7f91986d7700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91900f56e0 0x7f9190107bb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:58.279 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.276+0000 7f91986d7700 1 --2- 172.21.15.71:0/1600477268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f91900f6980 0x7f919010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:37:58.279 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.276+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 >> 172.21.15.71:0/1600477268 conn(0x7f91900f0e20 msgr2=0x7f91900fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:37:58.279 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.276+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 shutdown_connections 2021-11-20T07:37:58.279 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:37:58.276+0000 7f91986d7700 1 -- 172.21.15.71:0/1600477268 wait complete. 2021-11-20T07:37:58.734 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:58 smithi071 conmon[33687]: audit 2021-11-20T07:37:58.275324+0000 mon.smithi071 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.71:0/1600477268' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T07:37:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:58 smithi112 conmon[36062]: audit 2021-11-20T07:37:58.275324+0000 mon.smithi071 (mon.0) 438 : 2021-11-20T07:37:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:58 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/1600477268' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T07:37:58.834 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-11-20T07:37:58.834 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-11-20T07:37:58.835 INFO:tasks.cephadm:Setup complete, yielding 2021-11-20T07:37:58.835 INFO:teuthology.run_tasks:Running task print... 2021-11-20T07:37:58.848 INFO:teuthology.task.print:**** done starting v16.2.4 2021-11-20T07:37:58.849 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T07:37:58.877 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:37:58.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch status' 2021-11-20T07:37:59.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:59 smithi071 conmon[33687]: cluster 2021-11-20T07:37:58.358269+0000 mgr.smithi071.ynqmwn (mgr.14162) 169 2021-11-20T07:37:59.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:37:59 smithi071 conmon[33687]: : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:37:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:59 smithi112 conmon[36062]: cluster 2021-11-20T07:37:58.358269+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:37:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:37:59 smithi112 conmon[36062]: ) 169 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:00.303 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 -- 172.21.15.71:0/2408064839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f5440 msgr2=0x7fd3380f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:00.303 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 --2- 172.21.15.71:0/2408064839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f5440 0x7fd3380f5860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fd32c004660 tx=0x7fd32c009b30).stop 2021-11-20T07:38:00.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 -- 172.21.15.71:0/2408064839 shutdown_connections 2021-11-20T07:38:00.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 --2- 172.21.15.71:0/2408064839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f66e0 0x7fd3380f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:00.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 --2- 172.21.15.71:0/2408064839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f5440 0x7fd3380f5860 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:00.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 -- 172.21.15.71:0/2408064839 >> 172.21.15.71:0/2408064839 conn(0x7fd3380f0ba0 msgr2=0x7fd3380f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:00.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 -- 172.21.15.71:0/2408064839 shutdown_connections 2021-11-20T07:38:00.305 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.299+0000 7fd33df7c700 1 -- 172.21.15.71:0/2408064839 wait complete. 2021-11-20T07:38:00.305 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 Processor -- start 2021-11-20T07:38:00.305 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 -- start start 2021-11-20T07:38:00.305 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f5440 0x7fd338105d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:00.306 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:00.306 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd338106870 con 0x7fd3380f66e0 2021-11-20T07:38:00.306 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd33df7c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd338106fe0 con 0x7fd3380f5440 2021-11-20T07:38:00.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd337fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:00.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd337fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47750/0 (socket says 172.21.15.71:47750) 2021-11-20T07:38:00.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.300+0000 7fd337fff700 1 -- 172.21.15.71:0/116048705 learned_addr learned my addr 172.21.15.71:0/116048705 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:00.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd33cf7a700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f5440 0x7fd338105d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:00.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd337fff700 1 -- 172.21.15.71:0/116048705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f5440 msgr2=0x7fd338105d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:00.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd337fff700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f5440 0x7fd338105d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:00.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd337fff700 1 -- 172.21.15.71:0/116048705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c005040 con 0x7fd3380f66e0 2021-11-20T07:38:00.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd337fff700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd32400b410 tx=0x7fd324006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:00.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3240125f0 con 0x7fd3380f66e0 2021-11-20T07:38:00.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd324012c00 con 0x7fd3380f66e0 2021-11-20T07:38:00.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3381072c0 con 0x7fd3380f66e0 2021-11-20T07:38:00.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd338107880 con 0x7fd3380f66e0 2021-11-20T07:38:00.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.301+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd324002aa0 con 0x7fd3380f66e0 2021-11-20T07:38:00.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.302+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd32400f030 con 0x7fd3380f66e0 2021-11-20T07:38:00.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.302+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd338045ad0 con 0x7fd3380f66e0 2021-11-20T07:38:00.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.303+0000 7fd3357fa700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 0x7fd32805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:00.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.303+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd324076110 con 0x7fd3380f66e0 2021-11-20T07:38:00.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.303+0000 7fd33cf7a700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 0x7fd32805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:00.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.303+0000 7fd33cf7a700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 0x7fd32805e7c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd32c004e30 tx=0x7fd32c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:00.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.306+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd324049050 con 0x7fd3380f66e0 2021-11-20T07:38:00.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.487+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3380fddc0 con 0x7fd32805c300 2021-11-20T07:38:00.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.488+0000 7fd3357fa700 1 -- 172.21.15.71:0/116048705 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd3380fddc0 con 0x7fd32805c300 2021-11-20T07:38:00.491 INFO:teuthology.orchestra.run.smithi071.stdout:Backend: cephadm 2021-11-20T07:38:00.492 INFO:teuthology.orchestra.run.smithi071.stdout:Available: Yes 2021-11-20T07:38:00.492 INFO:teuthology.orchestra.run.smithi071.stdout:Paused: No 2021-11-20T07:38:00.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.489+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 msgr2=0x7fd32805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:00.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.489+0000 7fd33df7c700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 0x7fd32805e7c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd32c004e30 tx=0x7fd32c009b30).stop 2021-11-20T07:38:00.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.489+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 msgr2=0x7fd338106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:00.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.489+0000 7fd33df7c700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd32400b410 tx=0x7fd324006ef0).stop 2021-11-20T07:38:00.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 shutdown_connections 2021-11-20T07:38:00.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3380f5440 0x7fd338105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:00.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd32805c300 0x7fd32805e7c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:00.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 --2- 172.21.15.71:0/116048705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3380f66e0 0x7fd338106230 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:00.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 >> 172.21.15.71:0/116048705 conn(0x7fd3380f0ba0 msgr2=0x7fd3380fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:00.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 shutdown_connections 2021-11-20T07:38:00.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:00.490+0000 7fd33df7c700 1 -- 172.21.15.71:0/116048705 wait complete. 2021-11-20T07:38:00.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ps' 2021-11-20T07:38:01.580 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:01 smithi071 conmon[33687]: cluster 2021-11-20T07:38:00.358539+0000 mgr.smithi071.ynqmwn (mgr.14162) 170 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:01 smithi112 conmon[36062]: cluster 2021-11-20T07:38:00.358539+0000 mgr.smithi071.ynqmwn (mgr.14162) 170 : cluster [DBG] 2021-11-20T07:38:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:01 smithi112 conmon[36062]: pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:02.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- 172.21.15.71:0/426785346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f49a0 msgr2=0x7fb2500f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:02.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 --2- 172.21.15.71:0/426785346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f49a0 0x7fb2500f4dc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb24c004660 tx=0x7fb24c009b30).stop 2021-11-20T07:38:02.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- 172.21.15.71:0/426785346 shutdown_connections 2021-11-20T07:38:02.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 --2- 172.21.15.71:0/426785346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f5340 0x7fb2500f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 --2- 172.21.15.71:0/426785346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f49a0 0x7fb2500f4dc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- 172.21.15.71:0/426785346 >> 172.21.15.71:0/426785346 conn(0x7fb2500f04c0 msgr2=0x7fb2500f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:02.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- 172.21.15.71:0/426785346 shutdown_connections 2021-11-20T07:38:02.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- 172.21.15.71:0/426785346 wait complete. 2021-11-20T07:38:02.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 Processor -- start 2021-11-20T07:38:02.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.466+0000 7fb258429700 1 -- start start 2021-11-20T07:38:02.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f49a0 0x7fb2500f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:02.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:02.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2500f5f30 con 0x7fb2500f5340 2021-11-20T07:38:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2500f6070 con 0x7fb2500f49a0 2021-11-20T07:38:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47760/0 (socket says 172.21.15.71:47760) 2021-11-20T07:38:02.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 -- 172.21.15.71:0/1156138421 learned_addr learned my addr 172.21.15.71:0/1156138421 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:02.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2561c5700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f49a0 0x7fb2500f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:02.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 -- 172.21.15.71:0/1156138421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f49a0 msgr2=0x7fb2500f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:02.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f49a0 0x7fb2500f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 -- 172.21.15.71:0/1156138421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c005040 con 0x7fb2500f5340 2021-11-20T07:38:02.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb2559c4700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb24000c510 tx=0x7fb240007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:02.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb24000f650 con 0x7fb2500f5340 2021-11-20T07:38:02.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2500f6350 con 0x7fb2500f5340 2021-11-20T07:38:02.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb24000fc60 con 0x7fb2500f5340 2021-11-20T07:38:02.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb240012c80 con 0x7fb2500f5340 2021-11-20T07:38:02.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.467+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25010f010 con 0x7fb2500f5340 2021-11-20T07:38:02.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.468+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb250045ad0 con 0x7fb2500f5340 2021-11-20T07:38:02.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.469+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb240010070 con 0x7fb2500f5340 2021-11-20T07:38:02.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.469+0000 7fb246ffd700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 0x7fb23c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:02.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.469+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb2400758d0 con 0x7fb2500f5340 2021-11-20T07:38:02.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.469+0000 7fb2561c5700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 0x7fb23c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:02.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.469+0000 7fb2561c5700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 0x7fb23c05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb24c004e30 tx=0x7fb24c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:02.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.473+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb240048870 con 0x7fb2500f5340 2021-11-20T07:38:02.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:02 smithi071 conmon[33687]: audit 2021-11-20T07:38:00.489326+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:38:02.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:02 smithi071 conmon[33687]: mgr.14162) 171 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:02.652 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.651+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2500ff020 con 0x7fb23c05c2a0 2021-11-20T07:38:02.659 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.657+0000 7fb246ffd700 1 -- 172.21.15.71:0/1156138421 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2160 (secure 0 0 0) 0x7fb2500ff020 con 0x7fb23c05c2a0 2021-11-20T07:38:02.660 INFO:teuthology.orchestra.run.smithi071.stdout:NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID 2021-11-20T07:38:02.660 INFO:teuthology.orchestra.run.smithi071.stdout:alertmanager.smithi071 smithi071 *:9093,9094 running (2m) 41s ago 3m 0.20.0 0881eb8f169f 42825e1b70ee 2021-11-20T07:38:02.660 INFO:teuthology.orchestra.run.smithi071.stdout:crash.smithi071 smithi071 running (3m) 41s ago 3m 16.2.4 8d91d370c2b8 81d7564ae5bc 2021-11-20T07:38:02.662 INFO:teuthology.orchestra.run.smithi071.stdout:crash.smithi112 smithi112 running (2m) 50s ago 2m 16.2.4 8d91d370c2b8 01dca73db775 2021-11-20T07:38:02.662 INFO:teuthology.orchestra.run.smithi071.stdout:grafana.smithi071 smithi071 *:3000 running (2m) 41s ago 3m 6.7.4 557c83e11646 d3c060d0ff35 2021-11-20T07:38:02.662 INFO:teuthology.orchestra.run.smithi071.stdout:mgr.smithi071.ynqmwn smithi071 *:9283 running (4m) 41s ago 4m 16.2.4 8d91d370c2b8 1593c69cb4d1 2021-11-20T07:38:02.663 INFO:teuthology.orchestra.run.smithi071.stdout:mgr.smithi112.qaikrr smithi112 *:8443,9283 running (2m) 50s ago 2m 16.2.4 8d91d370c2b8 f1eb35e128a8 2021-11-20T07:38:02.663 INFO:teuthology.orchestra.run.smithi071.stdout:mon.smithi071 smithi071 running (4m) 41s ago 4m 16.2.4 8d91d370c2b8 95ec9c4b46c6 2021-11-20T07:38:02.664 INFO:teuthology.orchestra.run.smithi071.stdout:mon.smithi112 smithi112 running (2m) 50s ago 2m 16.2.4 8d91d370c2b8 b31f773ae07f 2021-11-20T07:38:02.664 INFO:teuthology.orchestra.run.smithi071.stdout:node-exporter.smithi071 smithi071 *:9100 running (3m) 41s ago 3m 0.18.1 e5a616e4b9cf c6a960532060 2021-11-20T07:38:02.664 INFO:teuthology.orchestra.run.smithi071.stdout:node-exporter.smithi112 smithi112 *:9100 running (2m) 50s ago 2m 0.18.1 e5a616e4b9cf a18ff6a249f7 2021-11-20T07:38:02.667 INFO:teuthology.orchestra.run.smithi071.stdout:osd.0 smithi071 running (2m) 41s ago 2m 16.2.4 8d91d370c2b8 95969e2fb38c 2021-11-20T07:38:02.667 INFO:teuthology.orchestra.run.smithi071.stdout:osd.1 smithi071 running (2m) 41s ago 2m 16.2.4 8d91d370c2b8 509fb5d169b3 2021-11-20T07:38:02.667 INFO:teuthology.orchestra.run.smithi071.stdout:osd.2 smithi071 running (104s) 41s ago 103s 16.2.4 8d91d370c2b8 66767378822e 2021-11-20T07:38:02.668 INFO:teuthology.orchestra.run.smithi071.stdout:osd.3 smithi112 running (88s) 50s ago 87s 16.2.4 8d91d370c2b8 873c2078da00 2021-11-20T07:38:02.668 INFO:teuthology.orchestra.run.smithi071.stdout:osd.4 smithi112 running (70s) 50s ago 70s 16.2.4 8d91d370c2b8 7cd688dc71c0 2021-11-20T07:38:02.668 INFO:teuthology.orchestra.run.smithi071.stdout:osd.5 smithi112 running (53s) 50s ago 53s 16.2.4 8d91d370c2b8 0380af612c96 2021-11-20T07:38:02.669 INFO:teuthology.orchestra.run.smithi071.stdout:prometheus.smithi071 smithi071 *:9095 running (2m) 41s ago 3m 2.18.1 de242295e225 54bf6e0399b5 2021-11-20T07:38:02.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 msgr2=0x7fb23c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:02.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 0x7fb23c05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb24c004e30 tx=0x7fb24c009b30).stop 2021-11-20T07:38:02.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 msgr2=0x7fb2500f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:02.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb24000c510 tx=0x7fb240007df0).stop 2021-11-20T07:38:02.674 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 shutdown_connections 2021-11-20T07:38:02.675 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2500f49a0 0x7fb2500f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.675 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb23c05c2a0 0x7fb23c05e760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.676 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 --2- 172.21.15.71:0/1156138421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb2500f5340 0x7fb2500f59d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:02.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 >> 172.21.15.71:0/1156138421 conn(0x7fb2500f04c0 msgr2=0x7fb2500fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:02.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 shutdown_connections 2021-11-20T07:38:02.678 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:02.659+0000 7fb258429700 1 -- 172.21.15.71:0/1156138421 wait complete. 2021-11-20T07:38:02.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:02 smithi112 conmon[36062]: audit 2021-11-20T07:38:00.489326+0000 mgr.smithi071.ynqmwn (mgr.14162) 171 : audit 2021-11-20T07:38:02.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:02 smithi112 conmon[36062]: [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:02.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ls' 2021-11-20T07:38:03.707 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:03 smithi071 conmon[33687]: cluster 2021-11-20T07:38:02.358999+0000 2021-11-20T07:38:03.708 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:03 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 172 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:03 smithi112 conmon[36062]: cluster 2021-11-20T07:38:02.358999+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:38:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:03 smithi112 conmon[36062]: ) 172 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:04.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:04 smithi071 conmon[33687]: audit 2021-11-20T07:38:02.652989+0000 mgr.smithi071.ynqmwn (mgr.14162) 173 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:04.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- 172.21.15.71:0/57813831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f6860 msgr2=0x7fe6440f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 --2- 172.21.15.71:0/57813831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f6860 0x7fe6440f6cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe62c004660 tx=0x7fe62c009b30).stop 2021-11-20T07:38:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- 172.21.15.71:0/57813831 shutdown_connections 2021-11-20T07:38:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 --2- 172.21.15.71:0/57813831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f6860 0x7fe6440f6cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:04.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 --2- 172.21.15.71:0/57813831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f55c0 0x7fe6440f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:04.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- 172.21.15.71:0/57813831 >> 172.21.15.71:0/57813831 conn(0x7fe6440f0d40 msgr2=0x7fe6440f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:04.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- 172.21.15.71:0/57813831 shutdown_connections 2021-11-20T07:38:04.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- 172.21.15.71:0/57813831 wait complete. 2021-11-20T07:38:04.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 Processor -- start 2021-11-20T07:38:04.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.574+0000 7fe64aec6700 1 -- start start 2021-11-20T07:38:04.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:04.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 0x7fe6441063c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:04.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe644106a00 con 0x7fe6440f55c0 2021-11-20T07:38:04.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441072b0 con 0x7fe6440f6860 2021-11-20T07:38:04.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:04.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47768/0 (socket says 172.21.15.71:47768) 2021-11-20T07:38:04.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 -- 172.21.15.71:0/1384007861 learned_addr learned my addr 172.21.15.71:0/1384007861 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:04.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe643fff700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 0x7fe6441063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:04.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 -- 172.21.15.71:0/1384007861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 msgr2=0x7fe6441063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:04.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 0x7fe6441063c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:04.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 -- 172.21.15.71:0/1384007861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c005040 con 0x7fe6440f55c0 2021-11-20T07:38:04.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe643fff700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 0x7fe6441063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:38:04.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe648c62700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe63400dd20 tx=0x7fe63400b550).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:04.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63400ede0 con 0x7fe6440f55c0 2021-11-20T07:38:04.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6340178d0 con 0x7fe6440f55c0 2021-11-20T07:38:04.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe644107530 con 0x7fe6440f55c0 2021-11-20T07:38:04.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6340168e0 con 0x7fe6440f55c0 2021-11-20T07:38:04.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.575+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe644107a90 con 0x7fe6440f55c0 2021-11-20T07:38:04.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.576+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6440f49b0 con 0x7fe6440f55c0 2021-11-20T07:38:04.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.577+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe634017a40 con 0x7fe6440f55c0 2021-11-20T07:38:04.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.577+0000 7fe6417fa700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 0x7fe63005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:04.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.577+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe634078a50 con 0x7fe6440f55c0 2021-11-20T07:38:04.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.577+0000 7fe643fff700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 0x7fe63005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:04.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.577+0000 7fe643fff700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 0x7fe63005e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe62c013580 tx=0x7fe62c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:04.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.581+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe63404bad0 con 0x7fe6440f55c0 2021-11-20T07:38:04.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.754+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6440ff8c0 con 0x7fe63005c2b0 2021-11-20T07:38:04.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.758+0000 7fe6417fa700 1 -- 172.21.15.71:0/1384007861 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1116 (secure 0 0 0) 0x7fe6440ff8c0 con 0x7fe63005c2b0 2021-11-20T07:38:04.761 INFO:teuthology.orchestra.run.smithi071.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2021-11-20T07:38:04.762 INFO:teuthology.orchestra.run.smithi071.stdout:alertmanager 1/1 43s ago 3m count:1 2021-11-20T07:38:04.762 INFO:teuthology.orchestra.run.smithi071.stdout:crash 2/2 52s ago 4m * 2021-11-20T07:38:04.762 INFO:teuthology.orchestra.run.smithi071.stdout:grafana 1/1 43s ago 3m count:1 2021-11-20T07:38:04.763 INFO:teuthology.orchestra.run.smithi071.stdout:mgr 2/2 52s ago 4m count:2 2021-11-20T07:38:04.763 INFO:teuthology.orchestra.run.smithi071.stdout:mon 2/2 52s ago 3m smithi071:172.21.15.71=smithi071;smithi112:172.21.15.112=smithi112;count:2 2021-11-20T07:38:04.763 INFO:teuthology.orchestra.run.smithi071.stdout:node-exporter 2/2 52s ago 3m * 2021-11-20T07:38:04.763 INFO:teuthology.orchestra.run.smithi071.stdout:osd.unmanaged 6/6 52s ago - 2021-11-20T07:38:04.764 INFO:teuthology.orchestra.run.smithi071.stdout:prometheus 1/1 43s ago 4m count:1 2021-11-20T07:38:04.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 msgr2=0x7fe63005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:04.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 0x7fe63005e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe62c013580 tx=0x7fe62c009b30).stop 2021-11-20T07:38:04.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 msgr2=0x7fe644105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:04.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe63400dd20 tx=0x7fe63400b550).stop 2021-11-20T07:38:04.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 shutdown_connections 2021-11-20T07:38:04.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe63005c2b0 0x7fe63005e770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:04.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6440f55c0 0x7fe644105ed0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:04.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 --2- 172.21.15.71:0/1384007861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6440f6860 0x7fe6441063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:04.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 >> 172.21.15.71:0/1384007861 conn(0x7fe6440f0d40 msgr2=0x7fe6440fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:04.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 shutdown_connections 2021-11-20T07:38:04.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:04.759+0000 7fe64aec6700 1 -- 172.21.15.71:0/1384007861 wait complete. 2021-11-20T07:38:04.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:04 smithi112 conmon[36062]: audit 2021-11-20T07:38:02.652989 2021-11-20T07:38:04.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:04 smithi112 conmon[36062]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 173 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:05.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch host ls' 2021-11-20T07:38:05.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:05 smithi071 conmon[33687]: cluster 2021-11-20T07:38:04.359436+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:05.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:05 smithi071 conmon[33687]: .14162) 174 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:05 smithi112 conmon[36062]: cluster 2021-11-20T07:38:04.359436+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:38:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:05 smithi112 conmon[36062]: 14162) 174 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:06.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 -- 172.21.15.71:0/317559852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 msgr2=0x7ff3680f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 --2- 172.21.15.71:0/317559852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff3680f6e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff358004660 tx=0x7ff358009b30).stop 2021-11-20T07:38:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 -- 172.21.15.71:0/317559852 shutdown_connections 2021-11-20T07:38:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 --2- 172.21.15.71:0/317559852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 0x7ff3680f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 --2- 172.21.15.71:0/317559852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff3680f6e30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 -- 172.21.15.71:0/317559852 >> 172.21.15.71:0/317559852 conn(0x7ff3680f0320 msgr2=0x7ff3680f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 -- 172.21.15.71:0/317559852 shutdown_connections 2021-11-20T07:38:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.462+0000 7ff36cc62700 1 -- 172.21.15.71:0/317559852 wait complete. 2021-11-20T07:38:06.469 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:06 smithi071 conmon[33687]: audit 2021-11-20T07:38:04.755983+0000 mgr.smithi071.ynqmwn (mgr.14162) 175 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:06.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 Processor -- start 2021-11-20T07:38:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 -- start start 2021-11-20T07:38:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 0x7ff368109470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368109a20 con 0x7ff3680f6a10 2021-11-20T07:38:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff36cc62700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368109b60 con 0x7ff3680f73b0 2021-11-20T07:38:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47778/0 (socket says 172.21.15.71:47778) 2021-11-20T07:38:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 -- 172.21.15.71:0/1044147712 learned_addr learned my addr 172.21.15.71:0/1044147712 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff366ffd700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 0x7ff368109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 -- 172.21.15.71:0/1044147712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 msgr2=0x7ff368109470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 0x7ff368109470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.463+0000 7ff3677fe700 1 -- 172.21.15.71:0/1044147712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358005040 con 0x7ff3680f6a10 2021-11-20T07:38:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff3677fe700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff358009830 tx=0x7ff358009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff358018070 con 0x7ff3680f6a10 2021-11-20T07:38:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36810a4a0 con 0x7ff3680f6a10 2021-11-20T07:38:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff358013410 con 0x7ff3680f6a10 2021-11-20T07:38:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35801c400 con 0x7ff3680f6a10 2021-11-20T07:38:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.464+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36810aa00 con 0x7ff3680f6a10 2021-11-20T07:38:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.465+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff368045ad0 con 0x7ff3680f6a10 2021-11-20T07:38:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.465+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff35801d420 con 0x7ff3680f6a10 2021-11-20T07:38:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.465+0000 7ff34ffff700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 0x7ff35005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.465+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff358077a30 con 0x7ff3680f6a10 2021-11-20T07:38:06.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.465+0000 7ff366ffd700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 0x7ff35005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:06.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.466+0000 7ff366ffd700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 0x7ff35005e7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff35c0041d0 tx=0x7ff35c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:06.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.469+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff35804aa30 con 0x7ff3680f6a10 2021-11-20T07:38:06.644 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.642+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff368101010 con 0x7ff35005c300 2021-11-20T07:38:06.645 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.644+0000 7ff34ffff700 1 -- 172.21.15.71:0/1044147712 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+126 (secure 0 0 0) 0x7ff368101010 con 0x7ff35005c300 2021-11-20T07:38:06.645 INFO:teuthology.orchestra.run.smithi071.stdout:HOST ADDR LABELS STATUS 2021-11-20T07:38:06.645 INFO:teuthology.orchestra.run.smithi071.stdout:smithi071 172.21.15.71 2021-11-20T07:38:06.646 INFO:teuthology.orchestra.run.smithi071.stdout:smithi112 smithi112 2021-11-20T07:38:06.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 msgr2=0x7ff35005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:06.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 0x7ff35005e7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff35c0041d0 tx=0x7ff35c009040).stop 2021-11-20T07:38:06.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 msgr2=0x7ff368108f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:06.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff358009830 tx=0x7ff358009b30).stop 2021-11-20T07:38:06.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 shutdown_connections 2021-11-20T07:38:06.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff35005c300 0x7ff35005e7c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3680f6a10 0x7ff368108f80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 --2- 172.21.15.71:0/1044147712 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff3680f73b0 0x7ff368109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:06.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 >> 172.21.15.71:0/1044147712 conn(0x7ff3680f0320 msgr2=0x7ff3680fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:06.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 shutdown_connections 2021-11-20T07:38:06.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:06.645+0000 7ff36cc62700 1 -- 172.21.15.71:0/1044147712 wait complete. 2021-11-20T07:38:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:06 smithi112 conmon[36062]: audit 2021-11-20T07:38:04.755983+0000 mgr.smithi071.ynqmwn (mgr.14162) 175 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:07.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch device ls' 2021-11-20T07:38:07.677 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:07 smithi071 conmon[33687]: cluster 2021-11-20T07:38:06.359787+0000 mgr.smithi071.ynqmwn (mgr.14162) 176 2021-11-20T07:38:07.678 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:07 smithi071 conmon[33687]: : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:07 smithi112 conmon[36062]: cluster 2021-11-20T07:38:06.359787+0000 2021-11-20T07:38:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:07 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 176 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:08.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 -- 172.21.15.71:0/1524754494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f66e0 msgr2=0x7f79640f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:08.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 --2- 172.21.15.71:0/1524754494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f66e0 0x7f79640f6b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7960004660 tx=0x7f7960009b30).stop 2021-11-20T07:38:08.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 -- 172.21.15.71:0/1524754494 shutdown_connections 2021-11-20T07:38:08.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 --2- 172.21.15.71:0/1524754494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f7980 0x7f79640f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 --2- 172.21.15.71:0/1524754494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f66e0 0x7f79640f6b00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 -- 172.21.15.71:0/1524754494 >> 172.21.15.71:0/1524754494 conn(0x7f79640f1e40 msgr2=0x7f79640f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:08.480 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 -- 172.21.15.71:0/1524754494 shutdown_connections 2021-11-20T07:38:08.480 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.475+0000 7f796c0f1700 1 -- 172.21.15.71:0/1524754494 wait complete. 2021-11-20T07:38:08.480 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:08 smithi071 conmon[33687]: audit 2021-11-20T07:38:06.644591+0000 mgr.smithi071.ynqmwn (mgr.14162) 177 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:08.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 Processor -- start 2021-11-20T07:38:08.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 -- start start 2021-11-20T07:38:08.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 0x7f7964106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:08.483 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:08.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7964107a70 con 0x7f79640f7980 2021-11-20T07:38:08.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796c0f1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7964107bb0 con 0x7f79640f66e0 2021-11-20T07:38:08.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:08.484 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47790/0 (socket says 172.21.15.71:47790) 2021-11-20T07:38:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 -- 172.21.15.71:0/2236450949 learned_addr learned my addr 172.21.15.71:0/2236450949 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f7969e8d700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 0x7f7964106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 -- 172.21.15.71:0/2236450949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 msgr2=0x7f7964106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 0x7f7964106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 -- 172.21.15.71:0/2236450949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7960005040 con 0x7f79640f7980 2021-11-20T07:38:08.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f7969e8d700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 0x7f7964106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:38:08.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.476+0000 7f796968c700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f795400c790 tx=0x7f7954007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:08.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.477+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795400e680 con 0x7f79640f7980 2021-11-20T07:38:08.486 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.477+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f795400ec90 con 0x7f79640f7980 2021-11-20T07:38:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.477+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7964108550 con 0x7f79640f7980 2021-11-20T07:38:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.477+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7964108b10 con 0x7f79640f7980 2021-11-20T07:38:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7954013d60 con 0x7f79640f7980 2021-11-20T07:38:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f795401a060 con 0x7f79640f7980 2021-11-20T07:38:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c004fa0 con 0x7f79640f7980 2021-11-20T07:38:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f795a7fc700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 0x7f795005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7954076de0 con 0x7f79640f7980 2021-11-20T07:38:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.478+0000 7f7969e8d700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 0x7f795005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.479+0000 7f7969e8d700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 0x7f795005e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7960009d60 tx=0x7f7960009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.482+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7954049d80 con 0x7f79640f7980 2021-11-20T07:38:08.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.656+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f794c0029f0 con 0x7f795005c2b0 2021-11-20T07:38:08.658 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.657+0000 7f795a7fc700 1 -- 172.21.15.71:0/2236450949 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7f794c0029f0 con 0x7f795005c2b0 2021-11-20T07:38:08.659 INFO:teuthology.orchestra.run.smithi071.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2021-11-20T07:38:08.659 INFO:teuthology.orchestra.run.smithi071.stdout:smithi071 /dev/nvme0n1 ssd PHFT62040172400BGN 400G Unknown N/A N/A No 2021-11-20T07:38:08.660 INFO:teuthology.orchestra.run.smithi071.stdout:smithi112 /dev/nvme0n1 ssd PHFT620400NV400BGN 400G Unknown N/A N/A No 2021-11-20T07:38:08.661 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 msgr2=0x7f795005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:08.661 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 0x7f795005e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7960009d60 tx=0x7f7960009b30).stop 2021-11-20T07:38:08.661 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 msgr2=0x7f79641074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:08.661 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f795400c790 tx=0x7f7954007d70).stop 2021-11-20T07:38:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 shutdown_connections 2021-11-20T07:38:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f79640f66e0 0x7f7964106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f795005c2b0 0x7f795005e770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 --2- 172.21.15.71:0/2236450949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79640f7980 0x7f79641074c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:08.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 >> 172.21.15.71:0/2236450949 conn(0x7f79640f1e40 msgr2=0x7f79640fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:08.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 shutdown_connections 2021-11-20T07:38:08.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:08.659+0000 7f796c0f1700 1 -- 172.21.15.71:0/2236450949 wait complete. 2021-11-20T07:38:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:08 smithi112 conmon[36062]: audit 2021-11-20T07:38:06.644591+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:08 smithi112 conmon[36062]: .14162) 177 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:09.035 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T07:38:09.052 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:38:09.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2021-11-20T07:38:09.710 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:09 smithi071 conmon[33687]: cluster 2021-11-20T07:38:08.360087+0000 mgr.smithi071.ynqmwn (mgr.14162) 178 2021-11-20T07:38:09.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:09 smithi071 conmon[33687]: : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:09 smithi112 conmon[36062]: cluster 2021-11-20T07:38:08.360087+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:09 smithi112 conmon[36062]: .14162) 178 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:10.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- 172.21.15.71:0/1744202488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 msgr2=0x7f393c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:10.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 --2- 172.21.15.71:0/1744202488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c0f69b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3930004660 tx=0x7f3930009b30).stop 2021-11-20T07:38:10.610 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- 172.21.15.71:0/1744202488 shutdown_connections 2021-11-20T07:38:10.610 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 --2- 172.21.15.71:0/1744202488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c0f69b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:10.610 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 --2- 172.21.15.71:0/1744202488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 0x7f393c0f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:10.610 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- 172.21.15.71:0/1744202488 >> 172.21.15.71:0/1744202488 conn(0x7f393c0f04c0 msgr2=0x7f393c0f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- 172.21.15.71:0/1744202488 shutdown_connections 2021-11-20T07:38:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- 172.21.15.71:0/1744202488 wait complete. 2021-11-20T07:38:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 Processor -- start 2021-11-20T07:38:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.604+0000 7f3944390700 1 -- start start 2021-11-20T07:38:10.611 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f3944390700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 0x7f393c105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:10.612 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f3944390700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:10.612 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f3944390700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1069c0 con 0x7f393c0f6550 2021-11-20T07:38:10.612 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f3944390700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1071d0 con 0x7f393c0f5bb0 2021-11-20T07:38:10.612 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:10.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47800/0 (socket says 172.21.15.71:47800) 2021-11-20T07:38:10.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 -- 172.21.15.71:0/1255287397 learned_addr learned my addr 172.21.15.71:0/1255287397 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:10.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394212c700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 0x7f393c105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:10.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 -- 172.21.15.71:0/1255287397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 msgr2=0x7f393c105ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:10.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 0x7f393c105ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:10.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 -- 172.21.15.71:0/1255287397 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930005040 con 0x7f393c0f6550 2021-11-20T07:38:10.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f394192b700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f3930009830 tx=0x7f3930009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:10.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3930018070 con 0x7f393c0f6550 2021-11-20T07:38:10.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3930013410 con 0x7f393c0f6550 2021-11-20T07:38:10.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.605+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393001c400 con 0x7f393c0f6550 2021-11-20T07:38:10.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.606+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c1073f0 con 0x7f393c0f6550 2021-11-20T07:38:10.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.606+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c107920 con 0x7f393c0f6550 2021-11-20T07:38:10.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.607+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c0f4150 con 0x7f393c0f6550 2021-11-20T07:38:10.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.607+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3930023020 con 0x7f393c0f6550 2021-11-20T07:38:10.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.607+0000 7f393affd700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 0x7f392405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:10.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.607+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f39300773b0 con 0x7f393c0f6550 2021-11-20T07:38:10.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.607+0000 7f394212c700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 0x7f392405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:10.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.608+0000 7f394212c700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 0x7f392405e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f393c105c40 tx=0x7f392c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:10.619 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.611+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f393004a3d0 con 0x7f393c0f6550 2021-11-20T07:38:10.619 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:10 smithi071 conmon[33687]: audit 2021-11-20T07:38:08.658036+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:38:10.619 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:10 smithi071 conmon[33687]: 14162) 179 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:10.619 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:10 smithi071 conmon[33687]: cluster 2021-11-20T07:38:10.360514+0000 mgr.smithi071.ynqmwn (mgr.14162) 180 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:10 smithi112 conmon[36062]: audit 2021-11-20T07:38:08.658036+0000 mgr.smithi071.ynqmwn (mgr.14162) 179 : audit [DBG] 2021-11-20T07:38:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:10 smithi112 conmon[36062]: from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:10 smithi112 conmon[36062]: cluster 2021-11-20T07:38:10.360514+0000 mgr.smithi071.ynqmwn (mgr.14162) 180 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:10.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:10.785+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 --> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f393c0fde80 con 0x7f392405c2b0 2021-11-20T07:38:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:11 smithi112 conmon[36062]: audit 2021-11-20T07:38:10.787033+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:38:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:11 smithi112 conmon[36062]: 14162) 181 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:11.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:11 smithi112 conmon[36062]: audit 2021-11-20T07:38:10.787520+0000 mon.smithi071 (mon.0) 439 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T07:38:11.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:11 smithi071 conmon[33687]: audit 2021-11-20T07:38:10.787033+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:11.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:11 smithi071 conmon[33687]: .14162) 181 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T07:38:11.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:11 smithi071 conmon[33687]: audit 2021-11-20T07:38:10.787520+0000 mon.smithi071 (mon.0) 439 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T07:38:12.430 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.428+0000 7f393affd700 1 -- 172.21.15.71:0/1255287397 <== mgr.14162 v2:172.21.15.71:6800/429188327 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f393c0fde80 con 0x7f392405c2b0 2021-11-20T07:38:12.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.430+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 msgr2=0x7f392405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:12.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.430+0000 7f3944390700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 0x7f392405e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f393c105c40 tx=0x7f392c009040).stop 2021-11-20T07:38:12.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.430+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 msgr2=0x7f393c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:12.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.430+0000 7f3944390700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f3930009830 tx=0x7f3930009b30).stop 2021-11-20T07:38:12.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.431+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 shutdown_connections 2021-11-20T07:38:12.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.431+0000 7f3944390700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0f5bb0 0x7f393c105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:12.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.431+0000 7f3944390700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f392405c2b0 0x7f392405e770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:12.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.431+0000 7f3944390700 1 --2- 172.21.15.71:0/1255287397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f393c0f6550 0x7f393c1063c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:12.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.431+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 >> 172.21.15.71:0/1255287397 conn(0x7f393c0f04c0 msgr2=0x7f393c0fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:12.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.432+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 shutdown_connections 2021-11-20T07:38:12.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:12.432+0000 7f3944390700 1 -- 172.21.15.71:0/1255287397 wait complete. 2021-11-20T07:38:12.702 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: audit 2021-11-20T07:38:11. 2021-11-20T07:38:12.703 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: 397961+0000 mon.smithi071 (mon.0) 440 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T07:38:12.703 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: cluster 2021-11-20T07:38:11.398006+0000 mon.smithi071 (mon.0) 441 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T07:38:12.703 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: audit 2021-11-20T07:38:11.399216+0000 mon.smithi071 (mon.0) 442 : audit [INF] 2021-11-20T07:38:12.703 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T07:38:12.704 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.704 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: 2021-11-20T07:38:12.360904+0000 mgr.smithi071.ynqmwn (mgr.14162) 182 : 2021-11-20T07:38:12.704 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: cluster [DBG] pgmap v144: 33 pgs: 13 creating+peering, 1 active+clean, 19 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:12.705 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:12 smithi071 conmon[33687]: debug 2021-11-20T07:38:12.407+0000 7f9c16b62700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T07:38:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: audit 2021-11-20T07:38:11.397961+0000 mon.smithi071 (mon 2021-11-20T07:38:12.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: .0) 440 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T07:38:12.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: cluster 2021-11-20T07:38:11.398006+0000 mon.smithi071 (mon.0) 441 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T07:38:12.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: audit 2021-11-20T07:38:11. 2021-11-20T07:38:12.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: 399216+0000 mon.smithi071 (mon.0) 442 : audit 2021-11-20T07:38:12.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T07:38:12.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: cluster 2021-11-20T07:38:12.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: 2021-11-20T07:38:12. 2021-11-20T07:38:12.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: 360904+0000 mgr.smithi071.ynqmwn (mgr.14162) 182 2021-11-20T07:38:12.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:12 smithi112 conmon[36062]: : cluster [DBG] pgmap v144: 33 pgs: 13 creating+peering, 1 active+clean, 19 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:12.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T07:38:13.727 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.404387+0000 mon.smithi071 (mon.0) 443 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T07:38:13.728 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.404572+0000 mon.smithi071 (mon.0) 444 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T07:38:13.728 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.408102+0000 mon.smithi071 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T07:38:13.728 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.408681+0000 mon.smithi071 (mon.0) 446 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T07:38:13.729 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.408698+0000 mon.smithi071 (mon.0) 447 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T07:38:13.729 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.418772+0000 mon.smithi071 (mon.0) 448 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T07:38:13.729 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.419814+0000 mon.smithi071 (mon.0) 449 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T07:38:13.730 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cluster 2021-11-20T07:38:12.420034+0000 mon.smithi071 (mon.0) 450 : cluster [DBG] fsmap cephfs:0 2021-11-20T07:38:13.730 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cephadm 2021-11-20T07:38:12.425970+0000 mgr.smithi071.ynqmwn (mgr.14162) 183 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T07:38:13.730 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:13.730 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: 2021-11-20T07:38:12.429558+0000 mon.smithi071 (mon.0) 451 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.432095+0000 mon.smithi071 (mon.0) 452 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:13.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.894815+0000 mon.smithi071 (mon.0) 453 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.905175+0000 mon.smithi071 (mon.0) 454 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.731 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.905903+0000 mon.smithi071 (mon.0) 455 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.bxcrgi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:13.732 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.912108+0000 mon.smithi071 (mon.0) 456 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.bxcrgi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:13.732 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: audit 2021-11-20T07:38:12.912793+0000 mon.smithi071 (mon.0) 457 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:13.732 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:13 smithi071 conmon[33687]: cephadm 2021-11-20T07:38:12.913431+0000 mgr.smithi071.ynqmwn (mgr.14162) 184 : cephadm [INF] Deploying daemon mds.cephfs.smithi112.bxcrgi on smithi112 2021-11-20T07:38:13.747 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T07:38:12.404387+0000 mon.smithi071 (mon.0) 443 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T07:38:13.748 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:38:13.748 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 11-20T07:38:12.404572+0000 mon.smithi071 (mon.0) 444 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T07:38:13.748 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T07:38:12.408102+0000 mon.smithi071 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T07:38:13.749 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cluster 2021-11-20T07:38:12.408681+0000 mon.smithi071 (mon.0) 446 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T07:38:13.749 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:38:13.749 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 07:38:12.408698+0000 mon.smithi071 (mon.0) 447 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T07:38:13.749 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cluster 2021-11-20T07:38:13.750 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38:12. 2021-11-20T07:38:13.750 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 418772+0000 mon.smithi071 (mon.0) 448 : cluster [DBG] 2021-11-20T07:38:13.750 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: osdmap e39: 6 total, 6 up, 6 in 2021-11-20T07:38:13.750 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:38:13.751 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 07:38:12.419814+0000 mon.smithi071 2021-11-20T07:38:13.751 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: (mon.0) 449 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T07:38:13.751 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38:13.751 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:38:13.751 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 07:38:12.420034+0000 mon.smithi071 (mon 2021-11-20T07:38:13.752 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: .0) 450 : cluster [DBG] fsmap cephfs:0 2021-11-20T07:38:13.752 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cephadm 2021-11-20T07:38:13.752 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07: 2021-11-20T07:38:13.752 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 38:12.425970+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:38:13.753 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 183 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T07:38:13.753 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:38:13.753 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 07:38:12.429558+0000 mon.smithi071 ( 2021-11-20T07:38:13.753 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: mon.0) 451 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.753 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T07:38:13.754 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38: 2021-11-20T07:38:13.754 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 12.432095+0000 mon.smithi071 (mon.0) 452 2021-11-20T07:38:13.754 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:13.754 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:38:13.755 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 20T07:38:12.894815+0000 2021-11-20T07:38:13.755 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: mon.smithi071 (mon.0) 453 : audit [INF] 2021-11-20T07:38:13.755 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.755 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:38:13.756 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 07:38:12.905175+0000 mon.smithi071 (mon. 2021-11-20T07:38:13.756 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 0) 454 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:13.756 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:38:13.756 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 11-20T07:38:12.905903+0000 2021-11-20T07:38:13.757 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: mon.smithi071 (mon.0) 455 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.bxcrgi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:13.757 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38:13.757 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20T07:38:13.757 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38:12 2021-11-20T07:38:13.757 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: .912108+0000 mon.smithi071 (mon.0) 456 : audit 2021-11-20T07:38:13.758 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.bxcrgi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:13.758 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:38:13.758 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: T07:38:12.912793+0000 mon.smithi071 2021-11-20T07:38:13.758 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: (mon.0) 457 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:13.759 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: cephadm 2021- 2021-11-20T07:38:13.759 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 11-20T07:38:12.913431+0000 2021-11-20T07:38:13.759 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 184 : cephadm [INF] Deploying daemon mds.cephfs.smithi112.bxcrgi on smithi112 2021-11-20T07:38:13.759 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:13 smithi112 conmon[36062]: 2021-11-20T07:38:14.333 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- 172.21.15.71:0/722896320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f4de0 msgr2=0x7f52000f5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:14.334 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 --2- 172.21.15.71:0/722896320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f4de0 0x7f52000f5200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f51f0004660 tx=0x7f51f0009b30).stop 2021-11-20T07:38:14.334 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- 172.21.15.71:0/722896320 shutdown_connections 2021-11-20T07:38:14.334 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 --2- 172.21.15.71:0/722896320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f5fe0 0x7f52000f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.334 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 --2- 172.21.15.71:0/722896320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f4de0 0x7f52000f5200 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.335 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- 172.21.15.71:0/722896320 >> 172.21.15.71:0/722896320 conn(0x7f52000f0540 msgr2=0x7f52000f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:14.335 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- 172.21.15.71:0/722896320 shutdown_connections 2021-11-20T07:38:14.335 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- 172.21.15.71:0/722896320 wait complete. 2021-11-20T07:38:14.335 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 Processor -- start 2021-11-20T07:38:14.336 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- start start 2021-11-20T07:38:14.336 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f4de0 0x7f5200106f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:14.336 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:14.336 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200107a10 con 0x7f52000f5fe0 2021-11-20T07:38:14.336 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f52048f9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200107b50 con 0x7f52000f4de0 2021-11-20T07:38:14.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.330+0000 7f51fdd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:14.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47810/0 (socket says 172.21.15.71:47810) 2021-11-20T07:38:14.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 -- 172.21.15.71:0/2070892914 learned_addr learned my addr 172.21.15.71:0/2070892914 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:14.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fe59c700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f4de0 0x7f5200106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 -- 172.21.15.71:0/2070892914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f4de0 msgr2=0x7f5200106f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f4de0 0x7f5200106f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 -- 172.21.15.71:0/2070892914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0005040 con 0x7f52000f5fe0 2021-11-20T07:38:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51fdd9b700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f51f4004870 tx=0x7f51f400c040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:14.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51f400f040 con 0x7f52000f5fe0 2021-11-20T07:38:14.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51f400fb10 con 0x7f52000f5fe0 2021-11-20T07:38:14.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52001084f0 con 0x7f52000f5fe0 2021-11-20T07:38:14.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51f4015b50 con 0x7f52000f5fe0 2021-11-20T07:38:14.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.331+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5200108ab0 con 0x7f52000f5fe0 2021-11-20T07:38:14.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.332+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5200045ad0 con 0x7f52000f5fe0 2021-11-20T07:38:14.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.332+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f51f401c060 con 0x7f52000f5fe0 2021-11-20T07:38:14.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.333+0000 7f51eeffd700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 0x7f51e805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:14.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.333+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f51f4075eb0 con 0x7f52000f5fe0 2021-11-20T07:38:14.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.333+0000 7f51fe59c700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 0x7f51e805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:14.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.333+0000 7f51fe59c700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 0x7f51e805e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f51f0009f50 tx=0x7f51f0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:14.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.337+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51f4048b70 con 0x7f52000f5fe0 2021-11-20T07:38:14.426 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:14 smithi112 conmon[36062]: cluster 2021-11-20T07:38 2021-11-20T07:38:14.427 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:14 smithi112 conmon[36062]: :13.418380+0000 mon.smithi071 (mon.0) 458 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T07:38:14.427 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:14 smithi112 conmon[36062]: cluster 2021-11-20T07:38:14.361360+0000 mgr.smithi071.ynqmwn (mgr.14162) 185 : cluster [DBG] pgmap v148: 65 pgs: 3 creating+activating, 13 creating+peering, 16 active+clean, 33 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:14.527 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.525+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f520010d590 con 0x7f52000f5fe0 2021-11-20T07:38:14.528 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.526+0000 7f51eeffd700 1 -- 172.21.15.71:0/2070892914 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7f51f404c190 con 0x7f52000f5fe0 2021-11-20T07:38:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:e2 2021-11-20T07:38:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T07:38:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:38:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:legacy client fscid: 1 2021-11-20T07:38:14.529 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Filesystem 'cephfs' (1) 2021-11-20T07:38:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:fs_name cephfs 2021-11-20T07:38:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 2 2021-11-20T07:38:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:flags 12 2021-11-20T07:38:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:created 2021-11-20T07:38:12.408568+0000 2021-11-20T07:38:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:modified 2021-11-20T07:38:12.408614+0000 2021-11-20T07:38:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:tableserver 0 2021-11-20T07:38:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:root 0 2021-11-20T07:38:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:session_timeout 60 2021-11-20T07:38:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:session_autoclose 300 2021-11-20T07:38:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:max_file_size 1099511627776 2021-11-20T07:38:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:required_client_features {} 2021-11-20T07:38:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:last_failure 0 2021-11-20T07:38:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:last_failure_osd_epoch 0 2021-11-20T07:38:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:38:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:max_mds 1 2021-11-20T07:38:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:in 2021-11-20T07:38:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:up {} 2021-11-20T07:38:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:failed 2021-11-20T07:38:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:damaged 2021-11-20T07:38:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:stopped 2021-11-20T07:38:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:data_pools [3] 2021-11-20T07:38:14.535 INFO:teuthology.orchestra.run.smithi071.stdout:metadata_pool 2 2021-11-20T07:38:14.535 INFO:teuthology.orchestra.run.smithi071.stdout:inline_data disabled 2021-11-20T07:38:14.535 INFO:teuthology.orchestra.run.smithi071.stdout:balancer 2021-11-20T07:38:14.535 INFO:teuthology.orchestra.run.smithi071.stdout:standby_count_wanted 0 2021-11-20T07:38:14.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:14.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:14.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 msgr2=0x7f51e805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:14.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 0x7f51e805e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f51f0009f50 tx=0x7f51f0009b30).stop 2021-11-20T07:38:14.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 msgr2=0x7f5200107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:14.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f51f4004870 tx=0x7f51f400c040).stop 2021-11-20T07:38:14.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 shutdown_connections 2021-11-20T07:38:14.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52000f4de0 0x7f5200106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f51e805c2b0 0x7f51e805e770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 --2- 172.21.15.71:0/2070892914 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52000f5fe0 0x7f5200107460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:14.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 >> 172.21.15.71:0/2070892914 conn(0x7f52000f0540 msgr2=0x7f52000f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:14.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 shutdown_connections 2021-11-20T07:38:14.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:14.528+0000 7f52048f9700 1 -- 172.21.15.71:0/2070892914 wait complete. 2021-11-20T07:38:14.540 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 2 2021-11-20T07:38:14.693 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:14 smithi071 conmon[33687]: cluster 2021-11-20T07:38: 2021-11-20T07:38:14.694 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:14 smithi071 conmon[33687]: 13.418380+0000 mon.smithi071 (mon.0) 458 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T07:38:14.694 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:14 smithi071 conmon[33687]: cluster 2021-11-20T07:38:14.361360+0000 mgr.smithi071.ynqmwn (mgr.14162) 185 : cluster [DBG] pgmap v148: 65 pgs: 3 creating+activating, 13 creating+peering, 16 active+clean, 33 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:14.990 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T07:38:15.000 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:38:15.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 2' 2021-11-20T07:38:15.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: cluster 2021-11-20T07:38:14.424606+0000 mon.smithi071 (mon.0) 459 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T07:38:15.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: audit 2021-11-20T07:38:14.527283+0000 mon.smithi071 (mon.0) 460 : audit [DBG] from='client.? 172.21.15.71:0/2070892914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T07:38:15.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: audit 2021-11-20T07:38:14.980039+0000 mon.smithi071 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:15.716 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: audit 2021-11-20T07:38:14.980843+0000 mon.smithi071 (mon.0) 462 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.xtyomb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:15.716 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: audit 2021-11-20T07:38:14.984165+0000 mon.smithi071 (mon.0) 463 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.xtyomb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:15.716 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: audit 2021-11-20T07:38:14.984991+0000 mon.smithi071 (mon.0) 464 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:15.716 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:15 smithi071 conmon[33687]: cephadm 2021-11-20T07:38:14.985608+0000 mgr.smithi071.ynqmwn (mgr.14162) 186 : cephadm [INF] Deploying daemon mds.cephfs.smithi071.xtyomb on smithi071 2021-11-20T07:38:15.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: cluster 2021-11-20T07:38:14.424606+0000 mon.smithi071 (mon.0 2021-11-20T07:38:15.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: ) 459 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T07:38:15.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: audit 2021-11-20T07:38:14.527283+0000 mon.smithi071 (mon.0) 460 : audit [DBG] from='client.? 172.21.15.71:0/2070892914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T07:38:15.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: audit 2021-11-20T07:38:14.980039+0000 mon.smithi071 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:15.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: audit 2021-11-20T07:38:14.980843+0000 mon.smithi071 (mon.0) 462 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.xtyomb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:15.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: audit 2021-11-20T07:38:14.984165+0000 mon.smithi071 (mon.0) 463 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.xtyomb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:15.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: audit 2021-11-20T07:38:14.984991+0000 mon.smithi071 (mon.0) 464 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:15.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:15 smithi112 conmon[36062]: cephadm 2021-11-20T07:38:14.985608+0000 mgr.smithi071.ynqmwn (mgr.14162) 186 : cephadm [INF] Deploying daemon mds.cephfs.smithi071.xtyomb on smithi071 2021-11-20T07:38:16.717 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:16 smithi071 conmon[33687]: cluster 2021-11-20T07:38:16.361807+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:38:16.718 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:16 smithi071 conmon[33687]: ) 187 : cluster [DBG] pgmap v150: 65 pgs: 3 creating+activating, 13 creating+peering, 31 active+clean, 18 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:16 smithi112 conmon[36062]: cluster 2021-11-20T07:38:16.361807+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:16 smithi112 conmon[36062]: .14162) 187 : cluster [DBG] pgmap v150: 65 pgs: 3 creating+activating, 13 creating+peering, 31 active+clean, 18 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:17.017 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 -- 172.21.15.71:0/3299679468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 msgr2=0x7fe6900f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:17.018 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 --2- 172.21.15.71:0/3299679468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe6900f4dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fe680004660 tx=0x7fe680009b30).stop 2021-11-20T07:38:17.018 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 -- 172.21.15.71:0/3299679468 shutdown_connections 2021-11-20T07:38:17.018 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 --2- 172.21.15.71:0/3299679468 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 0x7fe6900f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:17.018 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 --2- 172.21.15.71:0/3299679468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe6900f4dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:17.019 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.013+0000 7fe69802c700 1 -- 172.21.15.71:0/3299679468 >> 172.21.15.71:0/3299679468 conn(0x7fe6900f04c0 msgr2=0x7fe6900f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:17.019 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.014+0000 7fe69802c700 1 -- 172.21.15.71:0/3299679468 shutdown_connections 2021-11-20T07:38:17.019 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.014+0000 7fe69802c700 1 -- 172.21.15.71:0/3299679468 wait complete. 2021-11-20T07:38:17.019 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.014+0000 7fe69802c700 1 Processor -- start 2021-11-20T07:38:17.020 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe69802c700 1 -- start start 2021-11-20T07:38:17.020 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe69802c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:17.020 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe69802c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 0x7fe690107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:17.020 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe69802c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690107bb0 con 0x7fe6900f49a0 2021-11-20T07:38:17.021 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe69802c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690108460 con 0x7fe6900f5340 2021-11-20T07:38:17.021 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:17.021 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47818/0 (socket says 172.21.15.71:47818) 2021-11-20T07:38:17.021 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 -- 172.21.15.71:0/578185450 learned_addr learned my addr 172.21.15.71:0/578185450 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:17.022 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe6955c7700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 0x7fe690107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:17.022 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 -- 172.21.15.71:0/578185450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 msgr2=0x7fe690107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:17.022 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 0x7fe690107570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:17.023 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 -- 172.21.15.71:0/578185450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680005040 con 0x7fe6900f49a0 2021-11-20T07:38:17.023 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.015+0000 7fe695dc8700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe680009830 tx=0x7fe680009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:17.023 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.016+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe680018070 con 0x7fe6900f49a0 2021-11-20T07:38:17.023 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.016+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6901086e0 con 0x7fe6900f49a0 2021-11-20T07:38:17.025 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.016+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe680007ba0 con 0x7fe6900f49a0 2021-11-20T07:38:17.025 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.016+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe68001dea0 con 0x7fe6900f49a0 2021-11-20T07:38:17.025 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.016+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe690108c40 con 0x7fe6900f49a0 2021-11-20T07:38:17.025 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.017+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe680007d10 con 0x7fe6900f49a0 2021-11-20T07:38:17.026 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.017+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6900f4150 con 0x7fe6900f49a0 2021-11-20T07:38:17.026 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.017+0000 7fe6867fc700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 0x7fe67c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:17.026 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.018+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe680077ce0 con 0x7fe6900f49a0 2021-11-20T07:38:17.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.018+0000 7fe6955c7700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 0x7fe67c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:17.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.018+0000 7fe6955c7700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 0x7fe67c05e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe6900f5780 tx=0x7fe68c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:17.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.022+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe68004a9a0 con 0x7fe6900f49a0 2021-11-20T07:38:17.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.222+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fe690027a00 con 0x7fe6900f49a0 2021-11-20T07:38:17.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.431+0000 7fe6867fc700 1 -- 172.21.15.71:0/578185450 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v3) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe68004dfc0 con 0x7fe6900f49a0 2021-11-20T07:38:17.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 msgr2=0x7fe67c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:17.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 0x7fe67c05e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe6900f5780 tx=0x7fe68c008040).stop 2021-11-20T07:38:17.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 msgr2=0x7fe690107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:17.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe680009830 tx=0x7fe680009b30).stop 2021-11-20T07:38:17.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 shutdown_connections 2021-11-20T07:38:17.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe67c05c2b0 0x7fe67c05e770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:17.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6900f49a0 0x7fe690107080 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:17.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 --2- 172.21.15.71:0/578185450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6900f5340 0x7fe690107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:17.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 >> 172.21.15.71:0/578185450 conn(0x7fe6900f04c0 msgr2=0x7fe6900fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:17.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 shutdown_connections 2021-11-20T07:38:17.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:17.433+0000 7fe69802c700 1 -- 172.21.15.71:0/578185450 wait complete. 2021-11-20T07:38:17.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:17 smithi112 conmon[36062]: audit 2021-11-20T07:38:17.224491+0000 mon.smithi071 ( 2021-11-20T07:38:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:17 smithi112 conmon[36062]: mon.0) 465 : audit [INF] from='client.? 172.21.15.71:0/578185450' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-20T07:38:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:17 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.224491+0000 mon.smithi071 (mon.0) 465 : audit [INF] from='client.? 172.21.15.71:0/578185450' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-20T07:38:17.840 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T07:38:17.850 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:38:17.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2021-11-20T07:38:18.545 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.432530+0000 2021-11-20T07:38:18.546 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: mon.smithi071 (mon.0) 466 : audit [INF] from='client.? 172.21.15.71:0/578185450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-20T07:38:18.546 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: cluster 2021-11-20T07:38:17.432614+0000 mon.smithi071 (mon.0) 467 : cluster [DBG] fsmap cephfs:0 2021-11-20T07:38:18.546 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.596077+0000 mon.smithi071 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:18.547 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.596824+0000 mon.smithi071 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.iqrmps", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:18.547 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.599690+0000 mon.smithi071 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.iqrmps", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:18.547 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: audit 2021-11-20T07:38:17.600309+0000 mon.smithi071 (mon.0) 471 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:18.547 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: cephadm 2021-11-20T07:38:17.600879+0000 mgr.smithi071.ynqmwn (mgr.14162) 188 : cephadm [INF] Deploying daemon mds.cephfs.smithi112.iqrmps on smithi112 2021-11-20T07:38:18.548 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:18 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.362255+0000 mgr.smithi071.ynqmwn (mgr.14162) 189 : cluster [DBG] pgmap v151: 65 pgs: 3 creating+activating, 62 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:18.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: audit 2021-11-20T07:38:17.432530+0000 2021-11-20T07:38:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: mon.smithi071 (mon.0) 466 : audit [INF] from='client.? 172.21.15.71:0/578185450' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-20T07:38:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: cluster 2021-11-20T07:38:17.432614+0000 mon.smithi071 (mon.0) 467 : cluster [DBG] fsmap cephfs:0 2021-11-20T07:38:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: audit 2021-11-20T07:38:17.596077+0000 mon.smithi071 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:18.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: audit 2021-11-20T07:38:17.596824+0000 mon.smithi071 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.iqrmps", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:18.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: audit 2021-11-20T07:38:17.599690+0000 mon.smithi071 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.iqrmps", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:18.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:38:18.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: 20T07:38:17.600309+0000 2021-11-20T07:38:18.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: mon.smithi071 (mon.0) 471 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:18.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: 2021-11-20T07:38:18.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: cephadm 2021 2021-11-20T07:38:18.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: -11-20T07:38:17 2021-11-20T07:38:18.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: .600879+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:38:18.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: ) 188 : cephadm [INF] Deploying daemon mds.cephfs.smithi112.iqrmps on smithi112 2021-11-20T07:38:18.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: 2021-11-20T07:38:18.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: cluster 2021-11-20T07:38:18.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: 2021-11-20T07:38 2021-11-20T07:38:18.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: :18.362255+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:18.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: .14162) 189 : cluster [DBG] 2021-11-20T07:38:18.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:18 smithi112 conmon[36062]: pgmap v151: 65 pgs: 3 creating+activating, 62 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:19.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 -- 172.21.15.71:0/3709671819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 msgr2=0x7f52380f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:19.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 --2- 172.21.15.71:0/3709671819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f52380f6b40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f522c004660 tx=0x7f522c009b30).stop 2021-11-20T07:38:19.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 -- 172.21.15.71:0/3709671819 shutdown_connections 2021-11-20T07:38:19.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 --2- 172.21.15.71:0/3709671819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f52380f6b40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 --2- 172.21.15.71:0/3709671819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f52380f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 -- 172.21.15.71:0/3709671819 >> 172.21.15.71:0/3709671819 conn(0x7f52380f0ba0 msgr2=0x7f52380f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 -- 172.21.15.71:0/3709671819 shutdown_connections 2021-11-20T07:38:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.407+0000 7f5237fff700 1 -- 172.21.15.71:0/3709671819 wait complete. 2021-11-20T07:38:19.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 Processor -- start 2021-11-20T07:38:19.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 -- start start 2021-11-20T07:38:19.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f5238109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:19.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:19.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52380603e0 con 0x7f52380f66e0 2021-11-20T07:38:19.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5237fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238060520 con 0x7f52380f5440 2021-11-20T07:38:19.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:19.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47850/0 (socket says 172.21.15.71:47850) 2021-11-20T07:38:19.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 -- 172.21.15.71:0/976794706 learned_addr learned my addr 172.21.15.71:0/976794706 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5236ffd700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f5238109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 -- 172.21.15.71:0/976794706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 msgr2=0x7f5238109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f5238109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:19.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f52367fc700 1 -- 172.21.15.71:0/976794706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c005040 con 0x7f52380f66e0 2021-11-20T07:38:19.416 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.408+0000 7f5236ffd700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f5238109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:38:19.416 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f52367fc700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f522c009f50 tx=0x7f522c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:19.416 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522c018070 con 0x7f52380f66e0 2021-11-20T07:38:19.417 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f522c013410 con 0x7f52380f66e0 2021-11-20T07:38:19.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523810c500 con 0x7f52380f66e0 2021-11-20T07:38:19.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522c01c3c0 con 0x7f52380f66e0 2021-11-20T07:38:19.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.409+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523810ca60 con 0x7f52380f66e0 2021-11-20T07:38:19.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.410+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5238045ad0 con 0x7f52380f66e0 2021-11-20T07:38:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.410+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f522c01d420 con 0x7f52380f66e0 2021-11-20T07:38:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.410+0000 7f521f7fe700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 0x7f522005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.411+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f522c077bc0 con 0x7f52380f66e0 2021-11-20T07:38:19.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.411+0000 7f5236ffd700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 0x7f522005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:19.420 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.411+0000 7f5236ffd700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 0x7f522005e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5238060060 tx=0x7f5228009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:19.420 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.414+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f522c04a880 con 0x7f52380f66e0 2021-11-20T07:38:19.495 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021-11-20T07:38:18.434137+0000 mon.smithi071 (mon. 2021-11-20T07:38:19.495 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: 0) 472 : cluster [DBG] mds.? [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] up:boot 2021-11-20T07:38:19.495 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021-11-20T07:38:18.434202+0000 mon.smithi071 (mon.0) 473 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T07:38:19.495 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021-11-20T07:38:18.434316+0000 mon.smithi071 (mon.0) 474 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T07:38:19.496 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:38:19.496 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: -11-20T07:38:18.434550+0000 mon.smithi071 (mon.0) 475 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T07:38:19.496 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:38:19.496 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: T07:38:18.434644+0000 mon.smithi071 (mon.0) 476 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:38:19.497 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: 2021-11-20T07:38:19.497 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:38:19.497 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: -11-20T07:38:18.438875+0000 mon.smithi071 (mon.0 2021-11-20T07:38:19.497 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: ) 477 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi071.xtyomb=up:creating} 2021-11-20T07:38:19.497 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:38:19.498 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: :38:18.449291+0000 mon.smithi071 (mon.0) 478 : cluster 2021-11-20T07:38:19.498 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 0 2021-11-20T07:38:19.499 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.499 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: 2021-11-20T07:38:18.933491+0000 mon.smithi071 (mon.0) 2021-11-20T07:38:19.499 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: 479 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:38:19.499 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: audit 2021-11-20T07:38:18. 2021-11-20T07:38:19.500 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:19 smithi112 conmon[36062]: 933661+0000 mon.smithi071 (mon.0) 480 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:38:19.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:19.612+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5238027a00 con 0x7f52380f66e0 2021-11-20T07:38:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.434137+0000 mon.smithi071 ( 2021-11-20T07:38:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: mon.0) 472 : cluster [DBG] mds.? [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] up:boot 2021-11-20T07:38:19.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.434202+0000 mon.smithi071 (mon.0) 473 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T07:38:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.434316+0000 mon.smithi071 (mon.0) 474 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T07:38:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.434550+0000 mon.smithi071 (mon.0) 475 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T07:38:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: audit 2021-11-20T07:38:18.434644+0000 mon.smithi071 (mon.0) 476 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:38:19.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.438875+0000 mon.smithi071 (mon.0) 477 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi071.xtyomb=up:creating} 2021-11-20T07:38:19.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: cluster 2021-11-20T07:38:18.449291+0000 mon.smithi071 (mon.0) 478 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 0 2021-11-20T07:38:19.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: audit 2021-11-20T07:38:18.933491+0000 mon.smithi071 (mon.0) 479 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:38:19.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:19 smithi071 conmon[33687]: audit 2021-11-20T07:38:18.933661+0000 mon.smithi071 (mon.0) 480 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:38:20.448 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.446+0000 7f521f7fe700 1 -- 172.21.15.71:0/976794706 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5238027a00 con 0x7f52380f66e0 2021-11-20T07:38:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.448+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 msgr2=0x7f522005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.448+0000 7f5237fff700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 0x7f522005e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5238060060 tx=0x7f5228009c00).stop 2021-11-20T07:38:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 msgr2=0x7f523810bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f522c009f50 tx=0x7f522c009b30).stop 2021-11-20T07:38:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 shutdown_connections 2021-11-20T07:38:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52380f5440 0x7f5238109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f522005c300 0x7f522005e7c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 --2- 172.21.15.71:0/976794706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f52380f66e0 0x7f523810bfb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 >> 172.21.15.71:0/976794706 conn(0x7f52380f0ba0 msgr2=0x7f52380fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:20.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 shutdown_connections 2021-11-20T07:38:20.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:20.449+0000 7f5237fff700 1 -- 172.21.15.71:0/976794706 wait complete. 2021-11-20T07:38:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.438948+0000 mon.smithi071 (mon.0) 481 : cluster [DBG] mds.? [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] up:active 2021-11-20T07:38:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.439003+0000 mon.smithi071 (mon.0) 482 : cluster [DBG] mds.? [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] up:boot 2021-11-20T07:38:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.439058+0000 mon.smithi071 (mon.0) 483 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-20T07:38:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.439146+0000 mon.smithi071 (mon.0) 484 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:38:20.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.439165+0000 mon.smithi071 (mon.0) 485 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T07:38:20.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.439432+0000 mon.smithi071 (mon.0) 486 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:38:20.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.439554+0000 mon.smithi071 (mon.0) 487 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:38:20.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.442895+0000 mon.smithi071 (mon.0) 488 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:creating} 2021-11-20T07:38:20.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:19.455824+0000 mon.smithi071 (mon.0) 489 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:38:20.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.563611+0000 mon.smithi071 (mon.0) 490 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:20.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.564356+0000 mon.smithi071 (mon.0) 491 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.zaejme", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:20.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.566823+0000 mon.smithi071 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.zaejme", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:20.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.567429+0000 mon.smithi071 (mon.0) 493 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:20.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cephadm 2021-11-20T07:38:19.567981+0000 mgr.smithi071.ynqmwn (mgr.14162) 190 : cephadm 2021-11-20T07:38:20.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: [INF] Deploying daemon mds.cephfs.smithi071.zaejme on smithi071 2021-11-20T07:38:20.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: audit 2021-11-20T07:38:19.613992+0000 mon.smithi071 (mon.0) 494 : audit [INF] from='client.? 172.21.15.71:0/976794706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T07:38:20.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:20 smithi112 conmon[36062]: cluster 2021-11-20T07:38:20.362612+0000 mgr.smithi071.ynqmwn (mgr.14162) 191 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 182 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:20.790 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.438948+0000 2021-11-20T07:38:20.790 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: mon.smithi071 (mon.0) 481 : cluster [DBG] mds.? [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] up:active 2021-11-20T07:38:20.790 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.439003+0000 mon.smithi071 (mon.0) 482 : cluster [DBG] mds.? [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] up:boot 2021-11-20T07:38:20.790 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.439058+0000 mon.smithi071 (mon.0) 483 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-20T07:38:20.791 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.439146+0000 mon.smithi071 (mon.0) 484 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:38:20.791 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.439165+0000 mon.smithi071 (mon.0) 485 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T07:38:20.791 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.439432+0000 mon.smithi071 (mon.0) 486 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:38:20.791 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07:38:19.439554+0000 mon.smithi071 (mon.0) 487 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:38:20.792 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.442895+0000 mon.smithi071 (mon.0) 488 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:creating} 2021-11-20T07:38:20.792 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T07:38:19.455824+0000 mon.smithi071 (mon.0) 489 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:38:20.792 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07:38:19.563611+0000 mon.smithi071 (mon.0) 490 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:20.792 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07:38:19.564356+0000 mon.smithi071 (mon.0) 491 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.zaejme", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T07:38:20.793 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07:38:19.566823+0000 mon.smithi071 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi071.zaejme", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T07:38:20.793 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07:38:19.567429+0000 mon.smithi071 (mon.0) 493 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T07:38:20.793 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cephadm 2021-11-20T07:38: 2021-11-20T07:38:20.793 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: 19.567981+0000 mgr.smithi071.ynqmwn (mgr.14162) 190 : 2021-11-20T07:38:20.793 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cephadm [INF] Deploying daemon mds.cephfs.smithi071.zaejme on smithi071 2021-11-20T07:38:20.794 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: audit 2021-11-20T07 2021-11-20T07:38:20.794 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: :38:19.613992+0000 mon.smithi071 (mon 2021-11-20T07:38:20.794 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: .0) 494 : audit [INF] from='client.? 172.21.15.71:0/976794706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T07:38:20.794 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: 2021-11-20T07:38:20.795 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:38:20.795 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: 07:38:20.362612+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:38:20.795 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:20 smithi071 conmon[33687]: mgr.14162) 191 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 182 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:20.957 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T07:38:20.971 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:38:20.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T07:38:21.723 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:21 smithi071 conmon[33687]: cluster 2021-11-20T07:38:20.447501+0000 mon.smithi071 (mon.0) 495 : cluster [DBG] mds.? [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] up:active 2021-11-20T07:38:21.723 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:21 smithi071 conmon[33687]: audit 2021-11-20T07:38:20.447572+0000 mon.smithi071 (mon.0) 496 : audit [INF] from='client.? 172.21.15.71:0/976794706' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T07:38:21.724 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:21 smithi071 conmon[33687]: cluster 2021-11-20T07:38:20.447645+0000 mon.smithi071 (mon.0) 497 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:38:21.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:21 smithi112 conmon[36062]: cluster 2021-11-20T07:38:20.447501+0000 mon.smithi071 (mon.0) 495 : cluster [DBG] mds.? [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] up:active 2021-11-20T07:38:21.971 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:21 smithi112 conmon[36062]: audit 2021-11-20T07:38:20.447572+0000 mon.smithi071 (mon.0) 496 : audit [INF] from='client.? 172.21.15.71:0/976794706' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T07:38:21.972 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:21 smithi112 conmon[36062]: cluster 2021-11-20T07:38:20.447645+0000 mon.smithi071 (mon.0) 497 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:38:22.653 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.649+0000 7f17049f9700 1 -- 172.21.15.71:0/1466101200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 msgr2=0x7f17000f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:22.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.649+0000 7f17049f9700 1 --2- 172.21.15.71:0/1466101200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17000f59e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f16f0004660 tx=0x7f16f0009b30).stop 2021-11-20T07:38:22.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 -- 172.21.15.71:0/1466101200 shutdown_connections 2021-11-20T07:38:22.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 --2- 172.21.15.71:0/1466101200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 0x7f17000f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.655 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 --2- 172.21.15.71:0/1466101200 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17000f59e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.655 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 -- 172.21.15.71:0/1466101200 >> 172.21.15.71:0/1466101200 conn(0x7f17000f0d40 msgr2=0x7f17000f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:22.655 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 -- 172.21.15.71:0/1466101200 shutdown_connections 2021-11-20T07:38:22.656 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 -- 172.21.15.71:0/1466101200 wait complete. 2021-11-20T07:38:22.656 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 Processor -- start 2021-11-20T07:38:22.656 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 -- start start 2021-11-20T07:38:22.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:22.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.650+0000 7f17049f9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 0x7f17001096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:22.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f17049f9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700109c60 con 0x7f17000f55c0 2021-11-20T07:38:22.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f17049f9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700109da0 con 0x7f17000f6860 2021-11-20T07:38:22.657 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:22.658 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47870/0 (socket says 172.21.15.71:47870) 2021-11-20T07:38:22.658 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 -- 172.21.15.71:0/86912166 learned_addr learned my addr 172.21.15.71:0/86912166 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:22.658 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16f5d9b700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 0x7f17001096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:22.658 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 -- 172.21.15.71:0/86912166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 msgr2=0x7f17001096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:22.659 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 0x7f17001096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.659 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 -- 172.21.15.71:0/86912166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0005040 con 0x7f17000f55c0 2021-11-20T07:38:22.659 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16fe59c700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f16f0004e30 tx=0x7f16f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:22.659 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f0018070 con 0x7f17000f55c0 2021-11-20T07:38:22.660 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16f0013410 con 0x7f17000f55c0 2021-11-20T07:38:22.660 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170010a6e0 con 0x7f17000f55c0 2021-11-20T07:38:22.660 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f001de20 con 0x7f17000f55c0 2021-11-20T07:38:22.660 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.651+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170010ac40 con 0x7f17000f55c0 2021-11-20T07:38:22.661 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.653+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700045ad0 con 0x7f17000f55c0 2021-11-20T07:38:22.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.653+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f16f001d420 con 0x7f17000f55c0 2021-11-20T07:38:22.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.653+0000 7f16f77fe700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 0x7f16ec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:22.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.653+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f16f0077af0 con 0x7f17000f55c0 2021-11-20T07:38:22.663 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.653+0000 7f16f5d9b700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 0x7f16ec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:22.664 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.654+0000 7f16f5d9b700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 0x7f16ec05e760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f16e8004c40 tx=0x7f16e800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:22.664 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.657+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16f004a7b0 con 0x7f17000f55c0 2021-11-20T07:38:22.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.874+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f17000fcd80 con 0x7f17000f55c0 2021-11-20T07:38:22.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.874+0000 7f16f77fe700 1 -- 172.21.15.71:0/86912166 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 9 v9) v1 ==== 75+0+1464 (secure 0 0 0) 0x7f16f004ddd0 con 0x7f17000f55c0 2021-11-20T07:38:22.876 INFO:teuthology.orchestra.run.smithi071.stdout:e9 2021-11-20T07:38:22.877 INFO:teuthology.orchestra.run.smithi071.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T07:38:22.877 INFO:teuthology.orchestra.run.smithi071.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:38:22.877 INFO:teuthology.orchestra.run.smithi071.stdout:legacy client fscid: 1 2021-11-20T07:38:22.877 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:22.878 INFO:teuthology.orchestra.run.smithi071.stdout:Filesystem 'cephfs' (1) 2021-11-20T07:38:22.878 INFO:teuthology.orchestra.run.smithi071.stdout:fs_name cephfs 2021-11-20T07:38:22.878 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 8 2021-11-20T07:38:22.878 INFO:teuthology.orchestra.run.smithi071.stdout:flags 12 2021-11-20T07:38:22.878 INFO:teuthology.orchestra.run.smithi071.stdout:created 2021-11-20T07:38:12.408568+0000 2021-11-20T07:38:22.879 INFO:teuthology.orchestra.run.smithi071.stdout:modified 2021-11-20T07:38:20.440630+0000 2021-11-20T07:38:22.879 INFO:teuthology.orchestra.run.smithi071.stdout:tableserver 0 2021-11-20T07:38:22.879 INFO:teuthology.orchestra.run.smithi071.stdout:root 0 2021-11-20T07:38:22.879 INFO:teuthology.orchestra.run.smithi071.stdout:session_timeout 60 2021-11-20T07:38:22.880 INFO:teuthology.orchestra.run.smithi071.stdout:session_autoclose 300 2021-11-20T07:38:22.880 INFO:teuthology.orchestra.run.smithi071.stdout:max_file_size 1099511627776 2021-11-20T07:38:22.880 INFO:teuthology.orchestra.run.smithi071.stdout:required_client_features {} 2021-11-20T07:38:22.880 INFO:teuthology.orchestra.run.smithi071.stdout:last_failure 0 2021-11-20T07:38:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:last_failure_osd_epoch 0 2021-11-20T07:38:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T07:38:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:max_mds 2 2021-11-20T07:38:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:in 0,1 2021-11-20T07:38:22.881 INFO:teuthology.orchestra.run.smithi071.stdout:up {0=14430,1=24253} 2021-11-20T07:38:22.882 INFO:teuthology.orchestra.run.smithi071.stdout:failed 2021-11-20T07:38:22.882 INFO:teuthology.orchestra.run.smithi071.stdout:damaged 2021-11-20T07:38:22.882 INFO:teuthology.orchestra.run.smithi071.stdout:stopped 2021-11-20T07:38:22.882 INFO:teuthology.orchestra.run.smithi071.stdout:data_pools [3] 2021-11-20T07:38:22.883 INFO:teuthology.orchestra.run.smithi071.stdout:metadata_pool 2 2021-11-20T07:38:22.883 INFO:teuthology.orchestra.run.smithi071.stdout:inline_data disabled 2021-11-20T07:38:22.883 INFO:teuthology.orchestra.run.smithi071.stdout:balancer 2021-11-20T07:38:22.883 INFO:teuthology.orchestra.run.smithi071.stdout:standby_count_wanted 1 2021-11-20T07:38:22.883 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.cephfs.smithi071.xtyomb{0:14430} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204]] 2021-11-20T07:38:22.884 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.cephfs.smithi112.bxcrgi{1:24253} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819]] 2021-11-20T07:38:22.884 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:22.884 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:22.884 INFO:teuthology.orchestra.run.smithi071.stdout:Standby daemons: 2021-11-20T07:38:22.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:22.886 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.cephfs.smithi071.zaejme{-1:14442} state up:standby seq 1 addr [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265]] 2021-11-20T07:38:22.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 msgr2=0x7f16ec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:22.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 0x7f16ec05e760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f16e8004c40 tx=0x7f16e800b040).stop 2021-11-20T07:38:22.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 msgr2=0x7f17001091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:22.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f16f0004e30 tx=0x7f16f0009b30).stop 2021-11-20T07:38:22.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 shutdown_connections 2021-11-20T07:38:22.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f16ec05c2a0 0x7f16ec05e760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17000f55c0 0x7f17001091c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 --2- 172.21.15.71:0/86912166 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17000f6860 0x7f17001096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:22.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 >> 172.21.15.71:0/86912166 conn(0x7f17000f0d40 msgr2=0x7f17000fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:22.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 shutdown_connections 2021-11-20T07:38:22.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:22.876+0000 7f17049f9700 1 -- 172.21.15.71:0/86912166 wait complete. 2021-11-20T07:38:22.890 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 9 2021-11-20T07:38:23.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2021-11-20T07:38:23.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: cluster 2021-11-20T07:38:22.363014+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:38:23.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: 14162) 192 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s wr, 32 op/s 2021-11-20T07:38:23.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: audit 2021-11-20T07:38:22.376461+0000 mon.smithi071 (mon.0) 498 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:23.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: audit 2021-11-20T07:38:22.377126+0000 mon.smithi071 (mon.0) 499 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:23.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: cluster 2021-11-20T07:38:22.446810+0000 mon.smithi071 (mon.0) 500 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T07:38:23.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: cluster 2021-11-20T07:38:22.446837+0000 mon.smithi071 (mon.0) 501 : cluster [INF] Cluster is now healthy 2021-11-20T07:38:23.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: cluster 2021-11-20T07:38:22.452029+0000 mon.smithi071 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:boot 2021-11-20T07:38:23.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: cluster 2021-11-20T07:38:22.452093+0000 mon.smithi071 (mon.0) 503 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 1 up:standby 2021-11-20T07:38:23.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: audit 2021-11-20T07:38:22.452789+0000 mon.smithi071 (mon.0) 504 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:38:23.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:23 smithi071 conmon[33687]: audit 2021-11-20T07:38:22.875526+0000 mon.smithi071 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.71:0/86912166' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T07:38:23.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: cluster 2021-11-20T07:38:22. 2021-11-20T07:38:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: 363014+0000 mgr.smithi071.ynqmwn (mgr.14162) 192 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s wr, 32 op/s 2021-11-20T07:38:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: audit 2021-11-20T07:38:22.376461+0000 mon.smithi071 (mon.0) 498 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: audit 2021-11-20T07:38:22.377126+0000 mon.smithi071 (mon.0) 499 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: cluster 2021-11-20T07:38:22.446810+0000 mon.smithi071 (mon.0) 500 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T07:38:23.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: cluster 2021-11-20T07:38:22.446837+0000 mon.smithi071 (mon.0) 501 : cluster [INF] Cluster is now healthy 2021-11-20T07:38:23.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: cluster 2021-11-20T07:38:22.452029+0000 mon.smithi071 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:boot 2021-11-20T07:38:23.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: cluster 2021-11-20T07:38:22.452093+0000 mon.smithi071 (mon.0) 503 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 1 up:standby 2021-11-20T07:38:23.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: audit 2021-11-20T07:38:22.452789+0000 mon.smithi071 (mon.0) 504 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:38:23.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:23 smithi112 conmon[36062]: audit 2021-11-20T07:38:22.875526+0000 mon.smithi071 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.71:0/86912166' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T07:38:25.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.110+0000 7f476ca62700 1 -- 172.21.15.71:0/3239257253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 msgr2=0x7f47680f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:25.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.110+0000 7f476ca62700 1 --2- 172.21.15.71:0/3239257253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f47680f59e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f4758004660 tx=0x7f4758009b30).stop 2021-11-20T07:38:25.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- 172.21.15.71:0/3239257253 shutdown_connections 2021-11-20T07:38:25.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 --2- 172.21.15.71:0/3239257253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f47680f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 --2- 172.21.15.71:0/3239257253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f47680f59e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- 172.21.15.71:0/3239257253 >> 172.21.15.71:0/3239257253 conn(0x7f47680f0d40 msgr2=0x7f47680f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:25.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- 172.21.15.71:0/3239257253 shutdown_connections 2021-11-20T07:38:25.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- 172.21.15.71:0/3239257253 wait complete. 2021-11-20T07:38:25.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 Processor -- start 2021-11-20T07:38:25.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- start start 2021-11-20T07:38:25.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f476810c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47680603e0 con 0x7f47680f55c0 2021-11-20T07:38:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.111+0000 7f476ca62700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4768060520 con 0x7f47680f6860 2021-11-20T07:38:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:25.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47880/0 (socket says 172.21.15.71:47880) 2021-11-20T07:38:25.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 -- 172.21.15.71:0/30074907 learned_addr learned my addr 172.21.15.71:0/30074907 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:25.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f4765d9b700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f476810c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:25.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 -- 172.21.15.71:0/30074907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 msgr2=0x7f476810c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:25.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f476810c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 -- 172.21.15.71:0/30074907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4758005040 con 0x7f47680f55c0 2021-11-20T07:38:25.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f4765d9b700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f476810c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:38:25.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476659c700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4758004e30 tx=0x7f4758009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:25.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4758018070 con 0x7f47680f55c0 2021-11-20T07:38:25.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4758013410 con 0x7f47680f55c0 2021-11-20T07:38:25.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475801c3c0 con 0x7f47680f55c0 2021-11-20T07:38:25.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476810c6e0 con 0x7f47680f55c0 2021-11-20T07:38:25.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.112+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476810cc40 con 0x7f47680f55c0 2021-11-20T07:38:25.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.113+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4768045ad0 con 0x7f47680f55c0 2021-11-20T07:38:25.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.113+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f475801d420 con 0x7f47680f55c0 2021-11-20T07:38:25.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.114+0000 7f4756ffd700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 0x7f475005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:25.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.114+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4758077b40 con 0x7f47680f55c0 2021-11-20T07:38:25.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.114+0000 7f4765d9b700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 0x7f475005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:25.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.114+0000 7f4765d9b700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 0x7f475005e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f475c009740 tx=0x7f475c009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:25.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.118+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f475804a800 con 0x7f47680f55c0 2021-11-20T07:38:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.308+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47680fde90 con 0x7f47680f55c0 2021-11-20T07:38:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.308+0000 7f4756ffd700 1 -- 172.21.15.71:0/30074907 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f475804de20 con 0x7f47680f55c0 2021-11-20T07:38:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 msgr2=0x7f475005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 0x7f475005e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f475c009740 tx=0x7f475c009be0).stop 2021-11-20T07:38:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 msgr2=0x7f4768109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4758004e30 tx=0x7f4758009b30).stop 2021-11-20T07:38:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 shutdown_connections 2021-11-20T07:38:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f475005c2b0 0x7f475005e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47680f55c0 0x7f4768109c90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 --2- 172.21.15.71:0/30074907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47680f6860 0x7f476810c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 >> 172.21.15.71:0/30074907 conn(0x7f47680f0d40 msgr2=0x7f47680fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 shutdown_connections 2021-11-20T07:38:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:25.310+0000 7f476ca62700 1 -- 172.21.15.71:0/30074907 wait complete. 2021-11-20T07:38:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 10 2021-11-20T07:38:25.321 INFO:teuthology.orchestra.run.smithi071.stdout:true 2021-11-20T07:38:25.660 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: cluster 2021-11-20T07:38:24.363398+0000 mgr.smithi071.ynqmwn (mgr.14162) 193 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s wr, 18 op/s 2021-11-20T07:38:25.660 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: cluster 2021-11-20T07:38:24.381102+0000 mon.smithi071 (mon.0) 506 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:boot 2021-11-20T07:38:25.660 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: cluster 2021-11-20T07:38:24.381150+0000 mon.smithi071 (mon.0) 507 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:25.661 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: audit 2021-11-20T07:38:24.381248+0000 mon.smithi071 (mon.0) 508 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:38:25.661 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: audit 2021-11-20T07:38:25.247934+0000 mon.smithi071 (mon.0) 509 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:25.662 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:25 smithi071 conmon[33687]: audit 2021-11-20T07:38:25.309573+0000 mon.smithi071 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.71:0/30074907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:25.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2021-11-20T07:38:25.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: cluster 2021-11-20T07:38:24.363398+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:38:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: 193 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s wr, 18 op/s 2021-11-20T07:38:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: cluster 2021-11-20T07:38:24.381102+0000 mon.smithi071 (mon.0) 506 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:boot 2021-11-20T07:38:25.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: cluster 2021-11-20T07:38:24.381150+0000 mon.smithi071 ( 2021-11-20T07:38:25.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: mon.0) 507 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:25.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: audit 2021-11-20T07:38:25.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: 2021-11-20T07:38:24.381248+0000 mon.smithi071 ( 2021-11-20T07:38:25.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: mon.0) 508 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:38:25.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: audit 2021-11-20T07:38 2021-11-20T07:38:25.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: :25.247934+0000 mon.smithi071 (mon.0) 509 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:25.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:38:25.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: 07:38:25.309573+0000 mon.smithi071 (mon.0) 510 : audit [DBG] 2021-11-20T07:38:25.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:25 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/30074907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 -- 172.21.15.71:0/3498135591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f6530 msgr2=0x7fda6c0f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 --2- 172.21.15.71:0/3498135591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f6530 0x7fda6c0f6950 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fda68004660 tx=0x7fda68009b30).stop 2021-11-20T07:38:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 -- 172.21.15.71:0/3498135591 shutdown_connections 2021-11-20T07:38:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 --2- 172.21.15.71:0/3498135591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f77d0 0x7fda6c0f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 --2- 172.21.15.71:0/3498135591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f6530 0x7fda6c0f6950 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 -- 172.21.15.71:0/3498135591 >> 172.21.15.71:0/3498135591 conn(0x7fda6c0f1c90 msgr2=0x7fda6c0f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 -- 172.21.15.71:0/3498135591 shutdown_connections 2021-11-20T07:38:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.504+0000 7fda72ba1700 1 -- 172.21.15.71:0/3498135591 wait complete. 2021-11-20T07:38:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 Processor -- start 2021-11-20T07:38:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 -- start start 2021-11-20T07:38:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f6530 0x7fda6c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:27.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:27.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c06a380 con 0x7fda6c0f77d0 2021-11-20T07:38:27.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda72ba1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c06a4c0 con 0x7fda6c0f6530 2021-11-20T07:38:27.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:27.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47892/0 (socket says 172.21.15.71:47892) 2021-11-20T07:38:27.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 -- 172.21.15.71:0/16770631 learned_addr learned my addr 172.21.15.71:0/16770631 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:27.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda71b9f700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f6530 0x7fda6c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:27.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 -- 172.21.15.71:0/16770631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f6530 msgr2=0x7fda6c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:27.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f6530 0x7fda6c06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 -- 172.21.15.71:0/16770631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68005040 con 0x7fda6c0f77d0 2021-11-20T07:38:27.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.505+0000 7fda7139e700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fda5c0026d0 tx=0x7fda5c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:27.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.506+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda5c0079c0 con 0x7fda6c0f77d0 2021-11-20T07:38:27.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.506+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c06a7a0 con 0x7fda6c0f77d0 2021-11-20T07:38:27.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.506+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda5c00f040 con 0x7fda6c0f77d0 2021-11-20T07:38:27.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.506+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda5c0148b0 con 0x7fda6c0f77d0 2021-11-20T07:38:27.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.506+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c10ede0 con 0x7fda6c0f77d0 2021-11-20T07:38:27.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.507+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fda5c007b20 con 0x7fda6c0f77d0 2021-11-20T07:38:27.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.507+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c045ad0 con 0x7fda6c0f77d0 2021-11-20T07:38:27.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.507+0000 7fda627fc700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 0x7fda5805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:27.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.507+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fda5c075e40 con 0x7fda6c0f77d0 2021-11-20T07:38:27.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.507+0000 7fda71b9f700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 0x7fda5805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:27.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.508+0000 7fda71b9f700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 0x7fda5805e7c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fda6c06b730 tx=0x7fda68009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:27.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.511+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda5c048930 con 0x7fda6c0f77d0 2021-11-20T07:38:27.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.732+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fda6c027a00 con 0x7fda6c0f77d0 2021-11-20T07:38:27.734 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:27 smithi071 conmon[33687]: cluster 2021-11-20T07:38:26.363819+0000 mgr.smithi071.ynqmwn (mgr.14162) 194 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s wr, 12 op/s 2021-11-20T07:38:27.735 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:27 smithi071 conmon[33687]: cluster 2021-11-20T07:38:26.389349+0000 mon.smithi071 (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:standby 2021-11-20T07:38:27.735 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:27 smithi071 conmon[33687]: cluster 2021-11-20T07:38:26.389407+0000 mon.smithi071 (mon.0) 512 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:27.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.733+0000 7fda627fc700 1 -- 172.21.15.71:0/16770631 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v11) v1 ==== 78+0+86 (secure 0 0 0) 0x7fda5c04c120 con 0x7fda6c0f77d0 2021-11-20T07:38:27.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 msgr2=0x7fda5805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:27.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 0x7fda5805e7c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fda6c06b730 tx=0x7fda68009b30).stop 2021-11-20T07:38:27.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 msgr2=0x7fda6c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:27.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fda5c0026d0 tx=0x7fda5c009de0).stop 2021-11-20T07:38:27.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 shutdown_connections 2021-11-20T07:38:27.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c0f6530 0x7fda6c06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fda5805c300 0x7fda5805e7c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 --2- 172.21.15.71:0/16770631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda6c0f77d0 0x7fda6c069e20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:27.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.734+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 >> 172.21.15.71:0/16770631 conn(0x7fda6c0f1c90 msgr2=0x7fda6c0fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:27.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.735+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 shutdown_connections 2021-11-20T07:38:27.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:27.735+0000 7fda72ba1700 1 -- 172.21.15.71:0/16770631 wait complete. 2021-11-20T07:38:27.746 INFO:teuthology.orchestra.run.smithi071.stdout:true 2021-11-20T07:38:27.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:27 smithi112 conmon[36062]: cluster 2021-11-20T07:38:26.363819+0000 mgr.smithi071.ynqmwn (mgr.14162) 194 : cluster 2021-11-20T07:38:27.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:27 smithi112 conmon[36062]: [DBG] pgmap v155: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s wr, 12 op/s 2021-11-20T07:38:27.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:27 smithi112 conmon[36062]: cluster 2021-11-20T07:38:26.389349+0000 mon.smithi071 (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:standby 2021-11-20T07:38:27.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:27 smithi112 conmon[36062]: cluster 2021-11-20T07:38: 2021-11-20T07:38:27.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:27 smithi112 conmon[36062]: 26.389407+0000 mon.smithi071 (mon.0) 512 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:28.417 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2021-11-20T07:38:28.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T07:38:28.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- 172.21.15.71:0/4179088852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f6870 msgr2=0x7f82ec0f6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/4179088852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f6870 0x7f82ec0f6cd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f82dc004660 tx=0x7f82dc009b30).stop 2021-11-20T07:38:28.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- 172.21.15.71:0/4179088852 shutdown_connections 2021-11-20T07:38:28.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/4179088852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f6870 0x7f82ec0f6cd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/4179088852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ec0f55d0 0x7f82ec0f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- 172.21.15.71:0/4179088852 >> 172.21.15.71:0/4179088852 conn(0x7f82ec0f0d10 msgr2=0x7f82ec0f3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:28.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- 172.21.15.71:0/4179088852 shutdown_connections 2021-11-20T07:38:28.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- 172.21.15.71:0/4179088852 wait complete. 2021-11-20T07:38:28.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 Processor -- start 2021-11-20T07:38:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- start start 2021-11-20T07:38:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ec0f6870 0x7f82ec10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec10c1a0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.582+0000 7f82f3ee6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec10c2e0 con 0x7f82ec0f6870 2021-11-20T07:38:28.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:28.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47898/0 (socket says 172.21.15.71:47898) 2021-11-20T07:38:28.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 -- 172.21.15.71:0/2497634509 learned_addr learned my addr 172.21.15.71:0/2497634509 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:28.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 -- 172.21.15.71:0/2497634509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ec0f6870 msgr2=0x7f82ec10bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ec0f6870 0x7f82ec10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:28.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 -- 172.21.15.71:0/2497634509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc005040 con 0x7f82ec0f55d0 2021-11-20T07:38:28.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f1c82700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f82e800c790 tx=0x7f82e8007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:28.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e800e680 con 0x7f82ec0f55d0 2021-11-20T07:38:28.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82e800ec90 con 0x7f82ec0f55d0 2021-11-20T07:38:28.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec10c540 con 0x7f82ec0f55d0 2021-11-20T07:38:28.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e8012cb0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.583+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec10cad0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.584+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec046330 con 0x7f82ec0f55d0 2021-11-20T07:38:28.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.584+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f82e8010070 con 0x7f82ec0f55d0 2021-11-20T07:38:28.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.585+0000 7f82e27fc700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 0x7f82d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.585+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f82e8075fd0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.585+0000 7f82f1481700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 0x7f82d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:28.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.585+0000 7f82f1481700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 0x7f82d805e760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f82dc009f50 tx=0x7f82dc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:28.594 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.589+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82e8048c90 con 0x7f82ec0f55d0 2021-11-20T07:38:28.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.743+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82ec0586e0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.744+0000 7f82e27fc700 1 -- 172.21.15.71:0/2497634509 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14034 (secure 0 0 0) 0x7f82e804c2b0 con 0x7f82ec0f55d0 2021-11-20T07:38:28.747 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:28.747 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":41,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:38:14.416035+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T07:38:11.027597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T07:38:11.644741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:37:14.145361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:38:28.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 msgr2=0x7f82d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 0x7f82d805e760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f82dc009f50 tx=0x7f82dc009b30).stop 2021-11-20T07:38:28.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 msgr2=0x7f82ec05ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f82e800c790 tx=0x7f82e8007d90).stop 2021-11-20T07:38:28.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 shutdown_connections 2021-11-20T07:38:28.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f82d805c2a0 0x7f82d805e760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82ec0f55d0 0x7f82ec05ff20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 --2- 172.21.15.71:0/2497634509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82ec0f6870 0x7f82ec10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:28.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 >> 172.21.15.71:0/2497634509 conn(0x7f82ec0f0d10 msgr2=0x7f82ec0fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:28.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 shutdown_connections 2021-11-20T07:38:28.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.747+0000 7f82f3ee6700 1 -- 172.21.15.71:0/2497634509 wait complete. 2021-11-20T07:38:28.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:28 smithi071 conmon[33687]: audit 2021-11-20T07:38:27.734072+0000 mon.smithi071 (mon. 2021-11-20T07:38:28.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:28 smithi071 conmon[33687]: 0) 513 : audit [DBG] from='client.? 172.21.15.71:0/16770631' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T07:38:28.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T07:36:35.321439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T07:38:11.027597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T07:38:11.644741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T07:38:28.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T07:38:28.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:28 smithi112 conmon[36062]: audit 2021-11-20T07:38:27.734072+0000 mon.smithi071 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.71:0/16770631' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T07:38:28.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 -- 172.21.15.71:0/3150137690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 msgr2=0x7ff97c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 --2- 172.21.15.71:0/3150137690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0f59a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7ff978004660 tx=0x7ff978009b30).stop 2021-11-20T07:38:28.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 -- 172.21.15.71:0/3150137690 shutdown_connections 2021-11-20T07:38:28.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 --2- 172.21.15.71:0/3150137690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 --2- 172.21.15.71:0/3150137690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0f59a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.886+0000 7ff983500700 1 -- 172.21.15.71:0/3150137690 >> 172.21.15.71:0/3150137690 conn(0x7ff97c0f0ce0 msgr2=0x7ff97c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:28.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 -- 172.21.15.71:0/3150137690 shutdown_connections 2021-11-20T07:38:28.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 -- 172.21.15.71:0/3150137690 wait complete. 2021-11-20T07:38:28.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 Processor -- start 2021-11-20T07:38:28.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 -- start start 2021-11-20T07:38:28.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c10bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c10c130 con 0x7ff97c0f5580 2021-11-20T07:38:28.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.887+0000 7ff983500700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c10c270 con 0x7ff97c0f6820 2021-11-20T07:38:28.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:28.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47908/0 (socket says 172.21.15.71:47908) 2021-11-20T07:38:28.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 -- 172.21.15.71:0/3602480629 learned_addr learned my addr 172.21.15.71:0/3602480629 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:28.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff980a9b700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c10bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:28.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 -- 172.21.15.71:0/3602480629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 msgr2=0x7ff97c10bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:28.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c10bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 -- 172.21.15.71:0/3602480629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978005040 con 0x7ff97c0f5580 2021-11-20T07:38:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff980a9b700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c10bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:38:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff98129c700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff978009f50 tx=0x7ff978009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff978018070 con 0x7ff97c0f5580 2021-11-20T07:38:28.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff978013410 con 0x7ff97c0f5580 2021-11-20T07:38:28.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c10c4a0 con 0x7ff97c0f5580 2021-11-20T07:38:28.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97801c620 con 0x7ff97c0f5580 2021-11-20T07:38:28.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.888+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c10ca00 con 0x7ff97c0f5580 2021-11-20T07:38:28.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.889+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c004fa0 con 0x7ff97c0f5580 2021-11-20T07:38:28.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.890+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff97801c890 con 0x7ff97c0f5580 2021-11-20T07:38:28.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.890+0000 7ff971ffb700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 0x7ff96805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:28.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.890+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7ff978077d60 con 0x7ff97c0f5580 2021-11-20T07:38:28.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.890+0000 7ff980a9b700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 0x7ff96805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.890+0000 7ff980a9b700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 0x7ff96805e760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff96c0041d0 tx=0x7ff96c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:28.894+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff97804aa20 con 0x7ff97c0f5580 2021-11-20T07:38:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.052+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff95c005bf0 con 0x7ff97c0f5580 2021-11-20T07:38:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.052+0000 7ff971ffb700 1 -- 172.21.15.71:0/3602480629 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff97804e040 con 0x7ff97c0f5580 2021-11-20T07:38:29.054 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2021-11-20T07:38:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 msgr2=0x7ff96805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 0x7ff96805e760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff96c0041d0 tx=0x7ff96c009040).stop 2021-11-20T07:38:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 msgr2=0x7ff97c0604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff978009f50 tx=0x7ff978009b30).stop 2021-11-20T07:38:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 shutdown_connections 2021-11-20T07:38:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff96805c2a0 0x7ff96805e760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff97c0f5580 0x7ff97c0604c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 --2- 172.21.15.71:0/3602480629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff97c0f6820 0x7ff97c10bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 >> 172.21.15.71:0/3602480629 conn(0x7ff97c0f0ce0 msgr2=0x7ff97c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 shutdown_connections 2021-11-20T07:38:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.054+0000 7ff983500700 1 -- 172.21.15.71:0/3602480629 wait complete. 2021-11-20T07:38:29.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T07:38:29.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 -- 172.21.15.71:0/4087252672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f6820 msgr2=0x7f70380f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 --2- 172.21.15.71:0/4087252672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f6820 0x7f70380f6c80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7024004660 tx=0x7f7024009b30).stop 2021-11-20T07:38:29.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 -- 172.21.15.71:0/4087252672 shutdown_connections 2021-11-20T07:38:29.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 --2- 172.21.15.71:0/4087252672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f6820 0x7f70380f6c80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 --2- 172.21.15.71:0/4087252672 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f5580 0x7f70380f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 -- 172.21.15.71:0/4087252672 >> 172.21.15.71:0/4087252672 conn(0x7f70380f0ce0 msgr2=0x7f70380f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 -- 172.21.15.71:0/4087252672 shutdown_connections 2021-11-20T07:38:29.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.186+0000 7f703ea88700 1 -- 172.21.15.71:0/4087252672 wait complete. 2021-11-20T07:38:29.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 Processor -- start 2021-11-20T07:38:29.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 -- start start 2021-11-20T07:38:29.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f6820 0x7f703810bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703810c1a0 con 0x7f70380f5580 2021-11-20T07:38:29.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703ea88700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703810c2e0 con 0x7f70380f6820 2021-11-20T07:38:29.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47918/0 (socket says 172.21.15.71:47918) 2021-11-20T07:38:29.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 -- 172.21.15.71:0/3929536901 learned_addr learned my addr 172.21.15.71:0/3929536901 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:29.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f7037fff700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f6820 0x7f703810bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 -- 172.21.15.71:0/3929536901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f6820 msgr2=0x7f703810bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f6820 0x7f703810bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.187+0000 7f703c824700 1 -- 172.21.15.71:0/3929536901 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024005040 con 0x7f70380f5580 2021-11-20T07:38:29.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f703c824700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f702c002700 tx=0x7f702c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c0079c0 con 0x7f70380f5580 2021-11-20T07:38:29.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703810c540 con 0x7f70380f5580 2021-11-20T07:38:29.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f702c00f040 con 0x7f70380f5580 2021-11-20T07:38:29.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c014860 con 0x7f70380f5580 2021-11-20T07:38:29.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.188+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703810cad0 con 0x7f70380f5580 2021-11-20T07:38:29.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.189+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7038027a00 con 0x7f70380f5580 2021-11-20T07:38:29.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.189+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f702c01b060 con 0x7f70380f5580 2021-11-20T07:38:29.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.190+0000 7f70357fa700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.190+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f702c075fc0 con 0x7f70380f5580 2021-11-20T07:38:29.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.190+0000 7f7037fff700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.190+0000 7f7037fff700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 0x7f702805e760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7024009f50 tx=0x7f7024009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.193+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f702c048d70 con 0x7f70380f5580 2021-11-20T07:38:29.353 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.351+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f70380fdd10 con 0x7f70380f5580 2021-11-20T07:38:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.351+0000 7f70357fa700 1 -- 172.21.15.71:0/3929536901 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f702c04c390 con 0x7f70380f5580 2021-11-20T07:38:29.355 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 msgr2=0x7f702805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 0x7f702805e760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7024009f50 tx=0x7f7024009b30).stop 2021-11-20T07:38:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 msgr2=0x7f703805ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f702c002700 tx=0x7f702c009d80).stop 2021-11-20T07:38:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 shutdown_connections 2021-11-20T07:38:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f70380f5580 0x7f703805ff00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 --2- 172.21.15.71:0/3929536901 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70380f6820 0x7f703810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 >> 172.21.15.71:0/3929536901 conn(0x7f70380f0ce0 msgr2=0x7f70380fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 shutdown_connections 2021-11-20T07:38:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.353+0000 7f703ea88700 1 -- 172.21.15.71:0/3929536901 wait complete. 2021-11-20T07:38:29.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T07:38:29.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 -- 172.21.15.71:0/2454298551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40600e0 msgr2=0x7fc0f40f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2454298551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f9110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc0e4004660 tx=0x7fc0e4009b30).stop 2021-11-20T07:38:29.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 -- 172.21.15.71:0/2454298551 shutdown_connections 2021-11-20T07:38:29.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2454298551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40f9690 0x7fc0f40fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2454298551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f9110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.494 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.486+0000 7fc0f359e700 1 -- 172.21.15.71:0/2454298551 >> 172.21.15.71:0/2454298551 conn(0x7fc0f40f0b30 msgr2=0x7fc0f40f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 -- 172.21.15.71:0/2454298551 shutdown_connections 2021-11-20T07:38:29.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 -- 172.21.15.71:0/2454298551 wait complete. 2021-11-20T07:38:29.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 Processor -- start 2021-11-20T07:38:29.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 -- start start 2021-11-20T07:38:29.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40f5570 con 0x7fc0f40f9690 2021-11-20T07:38:29.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f359e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40f56b0 con 0x7fc0f40600e0 2021-11-20T07:38:29.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f1d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f1d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47930/0 (socket says 172.21.15.71:47930) 2021-11-20T07:38:29.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f1d9b700 1 -- 172.21.15.71:0/2414506547 learned_addr learned my addr 172.21.15.71:0/2414506547 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:29.498 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.487+0000 7fc0f259c700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.498 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f1d9b700 1 -- 172.21.15.71:0/2414506547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40600e0 msgr2=0x7fc0f40f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.498 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f1d9b700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f1d9b700 1 -- 172.21.15.71:0/2414506547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e4005040 con 0x7fc0f40f9690 2021-11-20T07:38:29.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f1d9b700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc0e800e7c0 tx=0x7fc0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e80092e0 con 0x7fc0f40f9690 2021-11-20T07:38:29.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f40f5990 con 0x7fc0f40f9690 2021-11-20T07:38:29.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f410ed30 con 0x7fc0f40f9690 2021-11-20T07:38:29.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e800f040 con 0x7fc0f40f9690 2021-11-20T07:38:29.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.488+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e8014750 con 0x7fc0f40f9690 2021-11-20T07:38:29.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.489+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d8004fa0 con 0x7fc0f40f9690 2021-11-20T07:38:29.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.489+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc0e80148d0 con 0x7fc0f40f9690 2021-11-20T07:38:29.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.489+0000 7fc0e2ffd700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 0x7fc0dc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.501 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.489+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc0e80760a0 con 0x7fc0f40f9690 2021-11-20T07:38:29.501 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.490+0000 7fc0f259c700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 0x7fc0dc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.501 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.490+0000 7fc0f259c700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 0x7fc0dc05e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc0e4013530 tx=0x7fc0e4016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.502 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.497+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0e8048d60 con 0x7fc0f40f9690 2021-11-20T07:38:29.668 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.666+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc0d8005bf0 con 0x7fc0f40f9690 2021-11-20T07:38:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.666+0000 7fc0e2ffd700 1 -- 172.21.15.71:0/2414506547 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc0e801d3f0 con 0x7fc0f40f9690 2021-11-20T07:38:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 msgr2=0x7fc0dc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 0x7fc0dc05e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc0e4013530 tx=0x7fc0e4016040).stop 2021-11-20T07:38:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 msgr2=0x7fc0f40f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc0e800e7c0 tx=0x7fc0e800b040).stop 2021-11-20T07:38:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 shutdown_connections 2021-11-20T07:38:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0f40600e0 0x7fc0f40f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc0dc05c2b0 0x7fc0dc05e770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 --2- 172.21.15.71:0/2414506547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0f40f9690 0x7fc0f40f5010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 >> 172.21.15.71:0/2414506547 conn(0x7fc0f40f0b30 msgr2=0x7fc0f40f2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 shutdown_connections 2021-11-20T07:38:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.668+0000 7fc0f359e700 1 -- 172.21.15.71:0/2414506547 wait complete. 2021-11-20T07:38:29.683 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: cluster 2021-11-20T07:38:28.364274+0000 mgr.smithi071.ynqmwn (mgr.14162) 195 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 9 op/s 2021-11-20T07:38:29.683 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: cluster 2021-11-20T07:38:28.395114+0000 mon.smithi071 (mon.0) 514 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:standby 2021-11-20T07:38:29.683 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: cluster 2021-11-20T07:38:28.395173+0000 mon.smithi071 (mon.0) 515 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:29.684 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:28.745352+0000 mon.smithi071 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.71:0/2497634509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:29.684 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:29.048549+0000 mon.smithi071 (mon.0) 517 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:29.684 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:29.051883+0000 mon.smithi071 (mon.0) 518 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:29.684 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:29.053698+0000 mon.smithi071 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.71:0/3602480629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:29.685 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:29.054925+0000 mon.smithi071 (mon.0) 520 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:29.685 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:29 smithi071 conmon[33687]: audit 2021-11-20T07:38:29.352740+0000 mon.smithi071 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.71:0/3929536901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:29.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:38:29.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: cluster 2021-11-20T07:38:28.364274+0000 mgr.smithi071.ynqmwn (mgr.14162) 195 : 2021-11-20T07:38:29.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 9 op/s 2021-11-20T07:38:29.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: cluster 2021-11-20T07:38:28.395114+0000 mon.smithi071 (mon.0) 514 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:standby 2021-11-20T07:38:29.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: cluster 2021-11-20T07:38:28.395173+0000 mon.smithi071 (mon.0) 515 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi071.xtyomb=up:active,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:29.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: 2021-11-20T07:38:29.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021-11- 2021-11-20T07:38:29.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: 20T07:38:28.745352+0000 mon.smithi071 (mon.0) 516 : audit [DBG] 2021-11-20T07:38:29.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/2497634509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:29.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021 2021-11-20T07:38:29.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: -11-20T07:38:29.048549+0000 mon.smithi071 (mon.0) 517 : audit 2021-11-20T07:38:29.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:29.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021-11-20T07:38:29.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: 2021-11-20T07:38:29.051883+0000 mon.smithi071 (mon.0) 518 2021-11-20T07:38:29.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:29.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021-11-20T07:38:29.053698+0000 2021-11-20T07:38:29.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: mon.smithi071 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.71:0/3602480629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:29.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021-11-20T07:38 2021-11-20T07:38:29.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: :29.054925+0000 mon.smithi071 (mon.0) 520 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:38:29.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: 2021-11-20T07:38:29.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:38:29.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:29 smithi112 conmon[36062]: 07:38:29.352740+0000 mon.smithi071 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.71:0/3929536901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:29.808 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 -- 172.21.15.71:0/1273902164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 msgr2=0x7f8c140f5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.808 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/1273902164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5f90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f8c04004660 tx=0x7f8c04009b30).stop 2021-11-20T07:38:29.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 -- 172.21.15.71:0/1273902164 shutdown_connections 2021-11-20T07:38:29.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/1273902164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 0x7f8c140f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/1273902164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5f90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 -- 172.21.15.71:0/1273902164 >> 172.21.15.71:0/1273902164 conn(0x7f8c140f0440 msgr2=0x7f8c140f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:29.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 -- 172.21.15.71:0/1273902164 shutdown_connections 2021-11-20T07:38:29.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.804+0000 7f8c1c28f700 1 -- 172.21.15.71:0/1273902164 wait complete. 2021-11-20T07:38:29.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 Processor -- start 2021-11-20T07:38:29.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 -- start start 2021-11-20T07:38:29.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 0x7f8c1410bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1410c120 con 0x7f8c140f5b70 2021-11-20T07:38:29.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1c28f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1410c260 con 0x7f8c140f6510 2021-11-20T07:38:29.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47938/0 (socket says 172.21.15.71:47938) 2021-11-20T07:38:29.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 -- 172.21.15.71:0/687177064 learned_addr learned my addr 172.21.15.71:0/687177064 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:29.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1982a700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 0x7f8c1410bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 -- 172.21.15.71:0/687177064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 msgr2=0x7f8c1410bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:29.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 0x7f8c1410bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:29.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.805+0000 7f8c1a02b700 1 -- 172.21.15.71:0/687177064 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04005040 con 0x7f8c140f5b70 2021-11-20T07:38:29.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c1a02b700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8c04005010 tx=0x7f8c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c04019070 con 0x7f8c140f5b70 2021-11-20T07:38:29.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c04007e70 con 0x7f8c140f5b70 2021-11-20T07:38:29.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1410c460 con 0x7f8c140f5b70 2021-11-20T07:38:29.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c04014d10 con 0x7f8c140f5b70 2021-11-20T07:38:29.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.806+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1410c9c0 con 0x7f8c140f5b70 2021-11-20T07:38:29.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.807+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14027a00 con 0x7f8c140f5b70 2021-11-20T07:38:29.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.807+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8c04025070 con 0x7f8c140f5b70 2021-11-20T07:38:29.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.808+0000 7f8c0affd700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 0x7f8c0005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:29.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.808+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8c04076ed0 con 0x7f8c140f5b70 2021-11-20T07:38:29.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.808+0000 7f8c1982a700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 0x7f8c0005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:29.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.808+0000 7f8c1982a700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 0x7f8c0005e760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8c10009fd0 tx=0x7f8c1000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:29.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:29.812+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c04049b90 con 0x7f8c140f5b70 2021-11-20T07:38:30.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.006+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c140fdd30 con 0x7f8c140f5b70 2021-11-20T07:38:30.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.007+0000 7f8c0affd700 1 -- 172.21.15.71:0/687177064 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f8c0404d1b0 con 0x7f8c140f5b70 2021-11-20T07:38:30.009 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:30.010 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/1338231265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1338231265},{"type":"v1","addr":"172.21.15.71:6829","nonce":1338231265}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24267,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3800926744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3800926744},{"type":"v1","addr":"172.21.15.112:6827","nonce":3800926744}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:38:20.440630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14430,"mds_1":24253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14430":{"gid":14430,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.71:6827/657608204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":657608204},{"type":"v1","addr":"172.21.15.71:6827","nonce":657608204}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24253":{"gid":24253,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":7,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6825/3641386819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3641386819},{"type":"v1","addr":"172.21.15.112:6825","nonce":3641386819}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:38:30.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 msgr2=0x7f8c0005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:30.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 0x7f8c0005e760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8c10009fd0 tx=0x7f8c1000b040).stop 2021-11-20T07:38:30.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 msgr2=0x7f8c140f5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8c04005010 tx=0x7f8c0400b040).stop 2021-11-20T07:38:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 shutdown_connections 2021-11-20T07:38:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8c0005c2a0 0x7f8c0005e760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8c140f5b70 0x7f8c140f5370 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 --2- 172.21.15.71:0/687177064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c140f6510 0x7f8c1410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 >> 172.21.15.71:0/687177064 conn(0x7f8c140f0440 msgr2=0x7f8c140fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 shutdown_connections 2021-11-20T07:38:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:30.009+0000 7f8c1c28f700 1 -- 172.21.15.71:0/687177064 wait complete. 2021-11-20T07:38:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 12 2021-11-20T07:38:30.024 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 8, 'max_mds': 2, 'flags': 18} 2021-11-20T07:38:30.024 INFO:teuthology.run_tasks:Running task kclient... 2021-11-20T07:38:30.055 INFO:tasks.kclient:Mounting kernel clients... 2021-11-20T07:38:30.056 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2021-11-20T07:38:30.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.057 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list 2021-11-20T07:38:30.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link delete ceph-brx 2021-11-20T07:38:30.142 INFO:teuthology.orchestra.run.smithi071.stderr:Cannot find device "ceph-brx" 2021-11-20T07:38:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:38:30.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.144 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2021-11-20T07:38:30.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.163 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link delete ceph-brx 2021-11-20T07:38:30.234 INFO:teuthology.orchestra.run.smithi112.stderr:Cannot find device "ceph-brx" 2021-11-20T07:38:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:38:30.236 INFO:tasks.kclient:client.0 config is {} 2021-11-20T07:38:30.236 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T07:38:30.237 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi071.front.sepia.ceph.com 2021-11-20T07:38:30.237 INFO:tasks.cephfs.mount:self.client.name = client.0 2021-11-20T07:38:30.238 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2021-11-20T07:38:30.239 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T07:38:30.239 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T07:38:30.239 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T07:38:30.240 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2021-11-20T07:38:30.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.240 DEBUG:teuthology.orchestra.run.smithi071:> ip addr 2021-11-20T07:38:30.258 INFO:teuthology.orchestra.run.smithi071.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T07:38:30.258 INFO:teuthology.orchestra.run.smithi071.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T07:38:30.258 INFO:teuthology.orchestra.run.smithi071.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T07:38:30.258 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:30.259 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 ::1/128 scope host 2021-11-20T07:38:30.259 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:30.259 INFO:teuthology.orchestra.run.smithi071.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:30.259 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4a brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:30.260 INFO:teuthology.orchestra.run.smithi071.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:30.260 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4b brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:30.260 INFO:teuthology.orchestra.run.smithi071.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:30.260 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:64 brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:30.261 INFO:teuthology.orchestra.run.smithi071.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T07:38:30.261 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:65 brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:30.261 INFO:teuthology.orchestra.run.smithi071.stdout: inet 172.21.15.71/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T07:38:30.261 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft 42260sec preferred_lft 42260sec 2021-11-20T07:38:30.262 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 fe80::ec4:7aff:fe8f:d765/64 scope link 2021-11-20T07:38:30.262 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:30.262 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T07:38:30.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.264 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2021-11-20T07:38:30.264 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2021-11-20T07:38:30.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add name ceph-brx type bridge 2021-11-20T07:38:30.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr flush dev ceph-brx 2021-11-20T07:38:30.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set ceph-brx up 2021-11-20T07:38:30.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T07:38:30.267 DEBUG:teuthology.orchestra.run.smithi071:> ') 2021-11-20T07:38:30.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.430 DEBUG:teuthology.orchestra.run.smithi071:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T07:38:30.457 INFO:teuthology.orchestra.run.smithi071.stdout:1 2021-11-20T07:38:30.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.459 DEBUG:teuthology.orchestra.run.smithi071:> route 2021-11-20T07:38:30.532 INFO:teuthology.orchestra.run.smithi071.stdout:Kernel IP routing table 2021-11-20T07:38:30.532 INFO:teuthology.orchestra.run.smithi071.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T07:38:30.532 INFO:teuthology.orchestra.run.smithi071.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T07:38:30.532 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T07:38:30.533 INFO:teuthology.orchestra.run.smithi071.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T07:38:30.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.534 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2021-11-20T07:38:30.534 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2021-11-20T07:38:30.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T07:38:30.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T07:38:30.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T07:38:30.535 DEBUG:teuthology.orchestra.run.smithi071:> ') 2021-11-20T07:38:30.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:30 smithi112 conmon[36062]: audit 2021-11-20T07:38:29.667785+0000 2021-11-20T07:38:30.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:30 smithi112 conmon[36062]: mon.smithi071 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.71:0/2414506547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:30.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:30 smithi112 conmon[36062]: audit 2021-11-20T07:38:30.008118+0000 mon.smithi071 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.71:0/687177064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:30.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:30 smithi112 conmon[36062]: cluster 2021-11-20T07:38:30.364661+0000 mgr.smithi071.ynqmwn (mgr.14162) 196 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 7 op/s 2021-11-20T07:38:30.608 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:30 smithi071 conmon[33687]: audit 2021-11-20T07:38:29. 2021-11-20T07:38:30.609 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:30 smithi071 conmon[33687]: 667785+0000 mon.smithi071 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.71:0/2414506547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:30.609 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:30 smithi071 conmon[33687]: audit 2021-11-20T07:38:30.008118+0000 mon.smithi071 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.71:0/687177064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:30.609 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:30 smithi071 conmon[33687]: cluster 2021-11-20T07:38:30.364661+0000 mgr.smithi071.ynqmwn (mgr.14162) 196 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 7 op/s 2021-11-20T07:38:30.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.877 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list 2021-11-20T07:38:30.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.935 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list-id 2021-11-20T07:38:30.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:30.991 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2021-11-20T07:38:30.991 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2021-11-20T07:38:30.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2021-11-20T07:38:30.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2021-11-20T07:38:30.992 DEBUG:teuthology.orchestra.run.smithi071:> ') 2021-11-20T07:38:31.083 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2021-11-20T07:38:31.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:31.084 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2021-11-20T07:38:31.084 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2021-11-20T07:38:31.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2021-11-20T07:38:31.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T07:38:31.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2021-11-20T07:38:31.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2021-11-20T07:38:31.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2021-11-20T07:38:31.087 DEBUG:teuthology.orchestra.run.smithi071:> ') 2021-11-20T07:38:31.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:31.308 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2021-11-20T07:38:31.308 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2021-11-20T07:38:31.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set brx.0 up 2021-11-20T07:38:31.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T07:38:31.309 DEBUG:teuthology.orchestra.run.smithi071:> ') 2021-11-20T07:38:31.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T07:38:31.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 -- 172.21.15.71:0/2164325024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f66a0 msgr2=0x7fbf1c0f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 --2- 172.21.15.71:0/2164325024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c0f6ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fbf140099e0 tx=0x7fbf14004d10).stop 2021-11-20T07:38:31.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 -- 172.21.15.71:0/2164325024 shutdown_connections 2021-11-20T07:38:31.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 --2- 172.21.15.71:0/2164325024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 --2- 172.21.15.71:0/2164325024 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c0f6ac0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 -- 172.21.15.71:0/2164325024 >> 172.21.15.71:0/2164325024 conn(0x7fbf1c0f1de0 msgr2=0x7fbf1c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:31.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 -- 172.21.15.71:0/2164325024 shutdown_connections 2021-11-20T07:38:31.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.560+0000 7fbf23e89700 1 -- 172.21.15.71:0/2164325024 wait complete. 2021-11-20T07:38:31.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 Processor -- start 2021-11-20T07:38:31.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 -- start start 2021-11-20T07:38:31.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c06bd70 con 0x7fbf1c0f7940 2021-11-20T07:38:31.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf23e89700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c06a310 con 0x7fbf1c0f66a0 2021-11-20T07:38:31.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47950/0 (socket says 172.21.15.71:47950) 2021-11-20T07:38:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 -- 172.21.15.71:0/1947763996 learned_addr learned my addr 172.21.15.71:0/1947763996 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:31.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21c25700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 -- 172.21.15.71:0/1947763996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f66a0 msgr2=0x7fbf1c06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 -- 172.21.15.71:0/1947763996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14009640 con 0x7fbf1c0f7940 2021-11-20T07:38:31.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf21424700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fbf0c00e7f0 tx=0x7fbf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:31.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.561+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf0c0092e0 con 0x7fbf1c0f7940 2021-11-20T07:38:31.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.562+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf0c00f040 con 0x7fbf1c0f7940 2021-11-20T07:38:31.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.562+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c06a590 con 0x7fbf1c0f7940 2021-11-20T07:38:31.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.562+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf0c014750 con 0x7fbf1c0f7940 2021-11-20T07:38:31.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.562+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c06ab50 con 0x7fbf1c0f7940 2021-11-20T07:38:31.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.562+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c027a00 con 0x7fbf1c0f7940 2021-11-20T07:38:31.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.563+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbf0c01b030 con 0x7fbf1c0f7940 2021-11-20T07:38:31.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.563+0000 7fbf127fc700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.563+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbf0c075bd0 con 0x7fbf1c0f7940 2021-11-20T07:38:31.574 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.563+0000 7fbf21c25700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.575 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.564+0000 7fbf21c25700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 0x7fbf0805e760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbf14003380 tx=0x7fbf14015040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:31.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.567+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf0c048940 con 0x7fbf1c0f7940 2021-11-20T07:38:31.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.726+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf1c0586e0 con 0x7fbf1c0f7940 2021-11-20T07:38:31.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.727+0000 7fbf127fc700 1 -- 172.21.15.71:0/1947763996 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14034 (secure 0 0 0) 0x7fbf0c019020 con 0x7fbf1c0f7940 2021-11-20T07:38:31.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:31.729 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":41,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:38:14.416035+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T07:38:11.027597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T07:38:11.644741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:37:14.145361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:38:31.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 msgr2=0x7fbf0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 0x7fbf0805e760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbf14003380 tx=0x7fbf14015040).stop 2021-11-20T07:38:31.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 msgr2=0x7fbf1c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fbf0c00e7f0 tx=0x7fbf0c00b040).stop 2021-11-20T07:38:31.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 shutdown_connections 2021-11-20T07:38:31.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf1c0f66a0 0x7fbf1c06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 --2- 172.21.15.71:0/1947763996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c0f7940 0x7fbf1c069e20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 >> 172.21.15.71:0/1947763996 conn(0x7fbf1c0f1de0 msgr2=0x7fbf1c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:31.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 shutdown_connections 2021-11-20T07:38:31.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.729+0000 7fbf23e89700 1 -- 172.21.15.71:0/1947763996 wait complete. 2021-11-20T07:38:31.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T07:36:35.321439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T07:38:11.027597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T07:38:11.644741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T07:38:31.744 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T07:38:31.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- 172.21.15.71:0/3926560663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f53d0 msgr2=0x7f14380f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 --2- 172.21.15.71:0/3926560663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f53d0 0x7f14380f57f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1434004660 tx=0x7f1434009b30).stop 2021-11-20T07:38:31.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- 172.21.15.71:0/3926560663 shutdown_connections 2021-11-20T07:38:31.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 --2- 172.21.15.71:0/3926560663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f6670 0x7f14380f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 --2- 172.21.15.71:0/3926560663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f53d0 0x7f14380f57f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- 172.21.15.71:0/3926560663 >> 172.21.15.71:0/3926560663 conn(0x7f14380f0b30 msgr2=0x7f14380f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:31.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- 172.21.15.71:0/3926560663 shutdown_connections 2021-11-20T07:38:31.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- 172.21.15.71:0/3926560663 wait complete. 2021-11-20T07:38:31.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 Processor -- start 2021-11-20T07:38:31.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.887+0000 7f143e0cd700 1 -- start start 2021-11-20T07:38:31.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143e0cd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f53d0 0x7f1438109ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143e0cd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143e0cd700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14380601f0 con 0x7f14380f6670 2021-11-20T07:38:31.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143e0cd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1438060330 con 0x7f14380f53d0 2021-11-20T07:38:31.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47960/0 (socket says 172.21.15.71:47960) 2021-11-20T07:38:31.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 -- 172.21.15.71:0/1278227428 learned_addr learned my addr 172.21.15.71:0/1278227428 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:31.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143d0cb700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f53d0 0x7f1438109ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 -- 172.21.15.71:0/1278227428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f53d0 msgr2=0x7f1438109ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:31.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f53d0 0x7f1438109ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:31.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 -- 172.21.15.71:0/1278227428 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1434005040 con 0x7f14380f6670 2021-11-20T07:38:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143c8ca700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f142c0026d0 tx=0x7f142c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f142c0079c0 con 0x7f14380f6670 2021-11-20T07:38:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f142c00f040 con 0x7f14380f6670 2021-11-20T07:38:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143810c590 con 0x7f14380f6670 2021-11-20T07:38:31.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.888+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f142c0148b0 con 0x7f14380f6670 2021-11-20T07:38:31.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.889+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143810cb20 con 0x7f14380f6670 2021-11-20T07:38:31.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.889+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14380f47c0 con 0x7f14380f6670 2021-11-20T07:38:31.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.890+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f142c007b20 con 0x7f14380f6670 2021-11-20T07:38:31.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.890+0000 7f1429ffb700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 0x7f142405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:31.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.890+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f142c075e40 con 0x7f14380f6670 2021-11-20T07:38:31.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.890+0000 7f143d0cb700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 0x7f142405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:31.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.890+0000 7f143d0cb700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 0x7f142405e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f143805fe70 tx=0x7f1434009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:31.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:31.894+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f142c048b00 con 0x7f14380f6670 2021-11-20T07:38:32.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.052+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1438045ad0 con 0x7f14380f6670 2021-11-20T07:38:32.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.053+0000 7f1429ffb700 1 -- 172.21.15.71:0/1278227428 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f142c04c120 con 0x7f14380f6670 2021-11-20T07:38:32.055 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2021-11-20T07:38:32.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 msgr2=0x7f142405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 0x7f142405e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f143805fe70 tx=0x7f1434009b30).stop 2021-11-20T07:38:32.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 msgr2=0x7f143810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f142c0026d0 tx=0x7f142c009de0).stop 2021-11-20T07:38:32.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 shutdown_connections 2021-11-20T07:38:32.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14380f53d0 0x7f1438109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f142405c2b0 0x7f142405e770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 --2- 172.21.15.71:0/1278227428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14380f6670 0x7f143810bfe0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 >> 172.21.15.71:0/1278227428 conn(0x7f14380f0b30 msgr2=0x7f14380fa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 shutdown_connections 2021-11-20T07:38:32.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.055+0000 7f143e0cd700 1 -- 172.21.15.71:0/1278227428 wait complete. 2021-11-20T07:38:32.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T07:38:32.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 -- 172.21.15.71:0/2710905600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 msgr2=0x7fc5f00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 --2- 172.21.15.71:0/2710905600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f00f59a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc5e0004660 tx=0x7fc5e0009b30).stop 2021-11-20T07:38:32.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 -- 172.21.15.71:0/2710905600 shutdown_connections 2021-11-20T07:38:32.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 --2- 172.21.15.71:0/2710905600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 0x7fc5f00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 --2- 172.21.15.71:0/2710905600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f00f59a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 -- 172.21.15.71:0/2710905600 >> 172.21.15.71:0/2710905600 conn(0x7fc5f00f0ce0 msgr2=0x7fc5f00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 -- 172.21.15.71:0/2710905600 shutdown_connections 2021-11-20T07:38:32.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 -- 172.21.15.71:0/2710905600 wait complete. 2021-11-20T07:38:32.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.189+0000 7fc5f6310700 1 Processor -- start 2021-11-20T07:38:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5f6310700 1 -- start start 2021-11-20T07:38:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5f6310700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5f6310700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 0x7fc5f010bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5f6310700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f010c1b0 con 0x7fc5f00f5580 2021-11-20T07:38:32.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5f6310700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f010c2f0 con 0x7fc5f00f6820 2021-11-20T07:38:32.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47968/0 (socket says 172.21.15.71:47968) 2021-11-20T07:38:32.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5ef7fe700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 0x7fc5f010bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 -- 172.21.15.71:0/83929280 learned_addr learned my addr 172.21.15.71:0/83929280 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:32.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 -- 172.21.15.71:0/83929280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 msgr2=0x7fc5f010bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 0x7fc5f010bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 -- 172.21.15.71:0/83929280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0005040 con 0x7fc5f00f5580 2021-11-20T07:38:32.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5effff700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc5e0004e30 tx=0x7fc5e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.190+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e0018070 con 0x7fc5f00f5580 2021-11-20T07:38:32.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.191+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f010c4f0 con 0x7fc5f00f5580 2021-11-20T07:38:32.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.191+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5e0013410 con 0x7fc5f00f5580 2021-11-20T07:38:32.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.191+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e001de20 con 0x7fc5f00f5580 2021-11-20T07:38:32.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.191+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f010ca20 con 0x7fc5f00f5580 2021-11-20T07:38:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.192+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f0046330 con 0x7fc5f00f5580 2021-11-20T07:38:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.192+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc5e001d420 con 0x7fc5f00f5580 2021-11-20T07:38:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.192+0000 7fc5ecff9700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 0x7fc5d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.192+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc5e0077af0 con 0x7fc5f00f5580 2021-11-20T07:38:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.192+0000 7fc5ef7fe700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 0x7fc5d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.193+0000 7fc5ef7fe700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 0x7fc5d805e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc5e4004c40 tx=0x7fc5e400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.196+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5e004a7b0 con 0x7fc5f00f5580 2021-11-20T07:38:32.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.357+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc5f0027a00 con 0x7fc5f00f5580 2021-11-20T07:38:32.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.357+0000 7fc5ecff9700 1 -- 172.21.15.71:0/83929280 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc5e004ddd0 con 0x7fc5f00f5580 2021-11-20T07:38:32.359 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:32.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 msgr2=0x7fc5d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 0x7fc5d805e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc5e4004c40 tx=0x7fc5e400b040).stop 2021-11-20T07:38:32.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 msgr2=0x7fc5f005ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc5e0004e30 tx=0x7fc5e0009b30).stop 2021-11-20T07:38:32.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 shutdown_connections 2021-11-20T07:38:32.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc5d805c2a0 0x7fc5d805e760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f00f5580 0x7fc5f005ff00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 --2- 172.21.15.71:0/83929280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5f00f6820 0x7fc5f010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 >> 172.21.15.71:0/83929280 conn(0x7fc5f00f0ce0 msgr2=0x7fc5f00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 shutdown_connections 2021-11-20T07:38:32.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.359+0000 7fc5f6310700 1 -- 172.21.15.71:0/83929280 wait complete. 2021-11-20T07:38:32.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T07:38:32.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 -- 172.21.15.71:0/2628446501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f6670 msgr2=0x7f323c0f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 --2- 172.21.15.71:0/2628446501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f6670 0x7f323c0f6ad0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f3228004660 tx=0x7f3228009b30).stop 2021-11-20T07:38:32.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 -- 172.21.15.71:0/2628446501 shutdown_connections 2021-11-20T07:38:32.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 --2- 172.21.15.71:0/2628446501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f6670 0x7f323c0f6ad0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 --2- 172.21.15.71:0/2628446501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f53d0 0x7f323c0f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 -- 172.21.15.71:0/2628446501 >> 172.21.15.71:0/2628446501 conn(0x7f323c0f0b30 msgr2=0x7f323c0f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 -- 172.21.15.71:0/2628446501 shutdown_connections 2021-11-20T07:38:32.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.501+0000 7f3241f7b700 1 -- 172.21.15.71:0/2628446501 wait complete. 2021-11-20T07:38:32.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 Processor -- start 2021-11-20T07:38:32.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 -- start start 2021-11-20T07:38:32.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 0x7f323c107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c107930 con 0x7f323c0f53d0 2021-11-20T07:38:32.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3241f7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c108140 con 0x7f323c0f6670 2021-11-20T07:38:32.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47978/0 (socket says 172.21.15.71:47978) 2021-11-20T07:38:32.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 -- 172.21.15.71:0/1989499498 learned_addr learned my addr 172.21.15.71:0/1989499498 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:32.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f323bfff700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 0x7f323c107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 -- 172.21.15.71:0/1989499498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 msgr2=0x7f323c107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 0x7f323c107330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 -- 172.21.15.71:0/1989499498 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228005040 con 0x7f323c0f53d0 2021-11-20T07:38:32.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f323bfff700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 0x7f323c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:38:32.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.502+0000 7f3240f79700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f323000c790 tx=0x7f3230007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f323000e680 con 0x7f323c0f53d0 2021-11-20T07:38:32.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c1083c0 con 0x7f323c0f53d0 2021-11-20T07:38:32.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f323000ec90 con 0x7f323c0f53d0 2021-11-20T07:38:32.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3230012cb0 con 0x7f323c0f53d0 2021-11-20T07:38:32.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c108980 con 0x7f323c0f53d0 2021-11-20T07:38:32.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.503+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c0f47c0 con 0x7f323c0f53d0 2021-11-20T07:38:32.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.504+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3230010070 con 0x7f323c0f53d0 2021-11-20T07:38:32.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.504+0000 7f32397fa700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 0x7f322c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.504+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f3230075fd0 con 0x7f323c0f53d0 2021-11-20T07:38:32.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.504+0000 7f323bfff700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 0x7f322c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.505+0000 7f323bfff700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 0x7f322c05e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3228013580 tx=0x7f3228009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.508+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3230048c90 con 0x7f323c0f53d0 2021-11-20T07:38:32.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.670+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f323c10d380 con 0x7f323c0f53d0 2021-11-20T07:38:32.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.670+0000 7f32397fa700 1 -- 172.21.15.71:0/1989499498 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f323004c2b0 con 0x7f323c0f53d0 2021-11-20T07:38:32.672 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:32.674 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 msgr2=0x7f322c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.674 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 0x7f322c05e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3228013580 tx=0x7f3228009b30).stop 2021-11-20T07:38:32.675 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 msgr2=0x7f323c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.676 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f323000c790 tx=0x7f3230007d90).stop 2021-11-20T07:38:32.676 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 shutdown_connections 2021-11-20T07:38:32.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f322c05c2b0 0x7f322c05e770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f323c0f53d0 0x7f323c106e40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 --2- 172.21.15.71:0/1989499498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f323c0f6670 0x7f323c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.677 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 >> 172.21.15.71:0/1989499498 conn(0x7f323c0f0b30 msgr2=0x7f323c05f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.678 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 shutdown_connections 2021-11-20T07:38:32.678 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.672+0000 7f3241f7b700 1 -- 172.21.15.71:0/1989499498 wait complete. 2021-11-20T07:38:32.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:32 smithi071 conmon[33687]: audit 2021-11-20T07:38:31.728318+0000 mon.smithi071 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.71:0/1947763996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:32.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:32 smithi071 conmon[33687]: audit 2021-11-20T07:38:31.922289+0000 mon.smithi071 (mon.0) 525 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:32.687 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:32 smithi071 conmon[33687]: audit 2021-11-20T07:38:32.054382+0000 mon.smithi071 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.71:0/1278227428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:32.687 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:32 smithi071 conmon[33687]: audit 2021-11-20T07:38:32.358851+0000 mon.smithi071 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.71:0/83929280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:32.688 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T07:38:32.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:38:32.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:32 smithi112 conmon[36062]: audit 2021-11-20T07:38:31.728318+0000 mon.smithi071 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.71:0/1947763996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:32 smithi112 conmon[36062]: audit 2021-11-20T07:38:31.922289+0000 mon.smithi071 (mon.0) 525 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:32.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:32 smithi112 conmon[36062]: audit 2021-11-20T07:38:32.054382+0000 mon.smithi071 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.71:0/1278227428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:32 smithi112 conmon[36062]: audit 2021-11-20T07:38:32.358851+0000 mon.smithi071 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.71:0/83929280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:32.808 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.804+0000 7f4251461700 1 -- 172.21.15.71:0/409929348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 msgr2=0x7f424c0f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.808 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.804+0000 7f4251461700 1 --2- 172.21.15.71:0/409929348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c0f58b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f423c004660 tx=0x7f423c009b30).stop 2021-11-20T07:38:32.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- 172.21.15.71:0/409929348 shutdown_connections 2021-11-20T07:38:32.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 --2- 172.21.15.71:0/409929348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 0x7f424c0f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.809 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 --2- 172.21.15.71:0/409929348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c0f58b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- 172.21.15.71:0/409929348 >> 172.21.15.71:0/409929348 conn(0x7f424c0f0bd0 msgr2=0x7f424c0f3010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:32.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- 172.21.15.71:0/409929348 shutdown_connections 2021-11-20T07:38:32.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- 172.21.15.71:0/409929348 wait complete. 2021-11-20T07:38:32.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 Processor -- start 2021-11-20T07:38:32.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- start start 2021-11-20T07:38:32.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 0x7f424c10bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c060210 con 0x7f424c0f5490 2021-11-20T07:38:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.805+0000 7f4251461700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c060350 con 0x7f424c0f6730 2021-11-20T07:38:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47988/0 (socket says 172.21.15.71:47988) 2021-11-20T07:38:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 -- 172.21.15.71:0/584742627 learned_addr learned my addr 172.21.15.71:0/584742627 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424a7fc700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 0x7f424c10bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 -- 172.21.15.71:0/584742627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 msgr2=0x7f424c10bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 0x7f424c10bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 -- 172.21.15.71:0/584742627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c005040 con 0x7f424c0f5490 2021-11-20T07:38:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f424affd700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f423c004e30 tx=0x7f423c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f423c018070 con 0x7f424c0f5490 2021-11-20T07:38:32.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f423c013410 con 0x7f424c0f5490 2021-11-20T07:38:32.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c10c4d0 con 0x7f424c0f5490 2021-11-20T07:38:32.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f423c01de20 con 0x7f424c0f5490 2021-11-20T07:38:32.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.806+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c10ca30 con 0x7f424c0f5490 2021-11-20T07:38:32.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.807+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424c045ad0 con 0x7f424c0f5490 2021-11-20T07:38:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.807+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f423c01d420 con 0x7f424c0f5490 2021-11-20T07:38:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.808+0000 7f42337fe700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 0x7f423405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.808+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f423c077b40 con 0x7f424c0f5490 2021-11-20T07:38:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.808+0000 7f424a7fc700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 0x7f423405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.808+0000 7f424a7fc700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 0x7f423405e7c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4240004c40 tx=0x7f424000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:32.812+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f423c04a800 con 0x7f424c0f5490 2021-11-20T07:38:33.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.004+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424c0fdd20 con 0x7f424c0f5490 2021-11-20T07:38:33.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.005+0000 7f42337fe700 1 -- 172.21.15.71:0/584742627 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f423c04de20 con 0x7f424c0f5490 2021-11-20T07:38:33.008 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:33.008 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/1338231265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1338231265},{"type":"v1","addr":"172.21.15.71:6829","nonce":1338231265}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24267,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3800926744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3800926744},{"type":"v1","addr":"172.21.15.112:6827","nonce":3800926744}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:38:20.440630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14430,"mds_1":24253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14430":{"gid":14430,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.71:6827/657608204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":657608204},{"type":"v1","addr":"172.21.15.71:6827","nonce":657608204}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24253":{"gid":24253,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":7,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6825/3641386819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3641386819},{"type":"v1","addr":"172.21.15.112:6825","nonce":3641386819}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:38:33.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 msgr2=0x7f423405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.010 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 0x7f423405e7c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4240004c40 tx=0x7f424000b040).stop 2021-11-20T07:38:33.010 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 msgr2=0x7f424c109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.010 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f423c004e30 tx=0x7f423c009b30).stop 2021-11-20T07:38:33.010 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 shutdown_connections 2021-11-20T07:38:33.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f423405c300 0x7f423405e7c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f424c0f5490 0x7f424c109a80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 --2- 172.21.15.71:0/584742627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f424c0f6730 0x7f424c10bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 >> 172.21.15.71:0/584742627 conn(0x7f424c0f0bd0 msgr2=0x7f424c0fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.011 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 shutdown_connections 2021-11-20T07:38:33.012 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.006+0000 7f4251461700 1 -- 172.21.15.71:0/584742627 wait complete. 2021-11-20T07:38:33.012 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 12 2021-11-20T07:38:33.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:38:20.440630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14430, 'mds_1': 24253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14430': {'gid': 14430, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.71:6827/657608204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 657608204}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 657608204}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24253': {'gid': 24253, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6825/3641386819', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 3641386819}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 3641386819}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:38:33.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-20T07:38:33.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi071.xtyomb status 2021-11-20T07:38:33.140 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- 172.21.15.71:0/719909034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f5590 msgr2=0x7fd8500f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.141 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 --2- 172.21.15.71:0/719909034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f5590 0x7fd8500f59b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fd8480099e0 tx=0x7fd848004d10).stop 2021-11-20T07:38:33.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- 172.21.15.71:0/719909034 shutdown_connections 2021-11-20T07:38:33.142 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 --2- 172.21.15.71:0/719909034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f6830 0x7fd8500f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.143 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 --2- 172.21.15.71:0/719909034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f5590 0x7fd8500f59b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.143 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- 172.21.15.71:0/719909034 >> 172.21.15.71:0/719909034 conn(0x7fd8500f0d10 msgr2=0x7fd8500f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.143 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- 172.21.15.71:0/719909034 shutdown_connections 2021-11-20T07:38:33.143 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- 172.21.15.71:0/719909034 wait complete. 2021-11-20T07:38:33.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 Processor -- start 2021-11-20T07:38:33.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- start start 2021-11-20T07:38:33.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f5590 0x7fd8500604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85010c270 con 0x7fd8500f6830 2021-11-20T07:38:33.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.136+0000 7fd854e54700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85010c3b0 con 0x7fd8500f5590 2021-11-20T07:38:33.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48000/0 (socket says 172.21.15.71:48000) 2021-11-20T07:38:33.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 -- 172.21.15.71:0/17632180 learned_addr learned my addr 172.21.15.71:0/17632180 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84e59c700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f5590 0x7fd8500604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 -- 172.21.15.71:0/17632180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f5590 msgr2=0x7fd8500604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f5590 0x7fd8500604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 -- 172.21.15.71:0/17632180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848009640 con 0x7fd8500f6830 2021-11-20T07:38:33.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fd8380026d0 tx=0x7fd838009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd846ffd700 1 -- 172.21.15.71:0/17632180 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8380079c0 con 0x7fd8500f6830 2021-11-20T07:38:33.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd846ffd700 1 -- 172.21.15.71:0/17632180 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd83800f040 con 0x7fd8500f6830 2021-11-20T07:38:33.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85010c610 con 0x7fd8500f6830 2021-11-20T07:38:33.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd846ffd700 1 -- 172.21.15.71:0/17632180 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8380148b0 con 0x7fd8500f6830 2021-11-20T07:38:33.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.137+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85010cb20 con 0x7fd8500f6830 2021-11-20T07:38:33.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.138+0000 7fd846ffd700 1 -- 172.21.15.71:0/17632180 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd838007b20 con 0x7fd8500f6830 2021-11-20T07:38:33.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.139+0000 7fd846ffd700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 0x7fd83c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.139+0000 7fd846ffd700 1 -- 172.21.15.71:0/17632180 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd838075e40 con 0x7fd8500f6830 2021-11-20T07:38:33.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.139+0000 7fd84e59c700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 0x7fd83c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.139+0000 7fd84e59c700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 0x7fd83c05e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd84800b620 tx=0x7fd848004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.140+0000 7fd8669ffb80 1 Processor -- start 2021-11-20T07:38:33.157 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd8669ffb80 1 -- start start 2021-11-20T07:38:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567ef076fe0 0x5567ef077d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eefb54d0 con 0x5567eeee41d0 2021-11-20T07:38:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eefc93e0 con 0x5567ef076fe0 2021-11-20T07:38:33.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48004/0 (socket says 172.21.15.71:48004) 2021-11-20T07:38:33.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 -- 172.21.15.71:0/4014833142 learned_addr learned my addr 172.21.15.71:0/4014833142 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.159 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567ef076fe0 0x5567ef077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 -- 172.21.15.71:0/4014833142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567ef076fe0 msgr2=0x5567ef077d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.160 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567ef076fe0 0x5567ef077d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 -- 172.21.15.71:0/4014833142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567ef0453c0 con 0x5567eeee41d0 2021-11-20T07:38:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd84e59c700 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fd84800b530 tx=0x7fd84801d000).ready entity=mon.0 client_cookie=1e1ad5206ed19d6d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.161 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd82ffff700 1 -- 172.21.15.71:0/4014833142 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd848024070 con 0x5567eeee41d0 2021-11-20T07:38:33.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd82ffff700 1 -- 172.21.15.71:0/4014833142 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd848019040 con 0x5567eeee41d0 2021-11-20T07:38:33.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.141+0000 7fd82ffff700 1 -- 172.21.15.71:0/4014833142 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd848013c90 con 0x5567eeee41d0 2021-11-20T07:38:33.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4014833142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 msgr2=0x5567ef076af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.162 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fd84800b530 tx=0x7fd84801d000).stop 2021-11-20T07:38:33.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4014833142 shutdown_connections 2021-11-20T07:38:33.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567ef076fe0 0x5567ef077d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4014833142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567eeee41d0 0x5567ef076af0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.163 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4014833142 >> 172.21.15.71:0/4014833142 conn(0x5567ef03c9b0 msgr2=0x5567ef071fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.164 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4014833142 shutdown_connections 2021-11-20T07:38:33.164 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4014833142 wait complete. 2021-11-20T07:38:33.164 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- start start 2021-11-20T07:38:33.164 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 Processor -- start 2021-11-20T07:38:33.164 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef083c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eefc9520 con 0x5567ef076fe0 2021-11-20T07:38:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.142+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eefc9aa0 con 0x5567eeee41d0 2021-11-20T07:38:33.165 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 10 client.0 ll_register_callbacks cb 0x5567ef036700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T07:38:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48010/0 (socket says 172.21.15.71:48010) 2021-11-20T07:38:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 -- 172.21.15.71:0/2666601488 learned_addr learned my addr 172.21.15.71:0/2666601488 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84e59c700 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef083c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.166 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 -- 172.21.15.71:0/2666601488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 msgr2=0x5567ef083c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.167 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef083c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.167 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 -- 172.21.15.71:0/2666601488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8500f7310 con 0x5567ef076fe0 2021-11-20T07:38:33.167 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fd838048cc0 tx=0x7fd83804f040).ready entity=mon.0 client_cookie=38ef93cd0de35df5 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.167 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2021-11-20T07:38:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd838014a10 con 0x5567ef076fe0 2021-11-20T07:38:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd838014b70 con 0x5567ef076fe0 2021-11-20T07:38:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5567ef044ce0 con 0x5567ef076fe0 2021-11-20T07:38:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd838014d30 con 0x5567ef076fe0 2021-11-20T07:38:33.168 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 10 client.14502 fetch_fsmap learned FSMap version 12 2021-11-20T07:38:33.169 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x5567eefb8630 con 0x5567ef076fe0 2021-11-20T07:38:33.169 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5567ef044ce0 con 0x5567ef076fe0 2021-11-20T07:38:33.169 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 12) v1 ==== 1736+0+0 (secure 0 0 0) 0x7fd83805f020 con 0x5567ef076fe0 2021-11-20T07:38:33.169 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 10 client.14502 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T07:38:33.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.143+0000 7fd8669ffb80 10 client.14502 resolve_mds: resolved name 'cephfs.smithi071.xtyomb' to daemon mds.cephfs.smithi071.xtyomb 2021-11-20T07:38:33.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd8669ffb80 20 client.14502 populate_metadata read hostname 'smithi071' 2021-11-20T07:38:33.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef08ca10 0x5567ef08ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.170 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd8669ffb80 4 client.14502 mds_command: new command op to 14430 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T07:38:33.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 --> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567eefbfdf0 con 0x5567ef08ca10 2021-11-20T07:38:33.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd84e59c700 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef08ca10 0x5567ef08ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd82d7fa700 10 client.14502 ms_handle_connect on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:33.171 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd84e59c700 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef08ca10 0x5567ef08ee00 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7fd2171a748f0884 server_cookie=c65638e1f40b7343 in_seq=0 out_seq=0 2021-11-20T07:38:33.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd82d7fa700 1 -- 172.21.15.71:0/2666601488 <== mds.0 v2:172.21.15.71:6826/657608204 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x5567eefbfdf0 con 0x5567ef08ca10 2021-11-20T07:38:33.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd82d7fa700 10 client.14502 handle_command_reply: tid=0 2021-11-20T07:38:33.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.144+0000 7fd8669ffb80 1 client.14502 shutdown 2021-11-20T07:38:33.172 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 msgr2=0x5567ef084160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fd838048cc0 tx=0x7fd83804f040).stop 2021-11-20T07:38:33.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 shutdown_connections 2021-11-20T07:38:33.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef08ca10 0x5567ef08ee00 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef083c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd82d7fa700 0 client.14502 ms_handle_reset on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:33.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/2666601488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef084160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 >> 172.21.15.71:0/2666601488 conn(0x5567ef03c9b0 msgr2=0x5567ef0731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 shutdown_connections 2021-11-20T07:38:33.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 1 -- 172.21.15.71:0/2666601488 wait complete. 2021-11-20T07:38:33.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.145+0000 7fd8669ffb80 20 client.14502 trim_cache size 0 max 16384 2021-11-20T07:38:33.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 Processor -- start 2021-11-20T07:38:33.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 -- start start 2021-11-20T07:38:33.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef082950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eef974a0 con 0x5567ef076fe0 2021-11-20T07:38:33.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.162+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567eefa9ab0 con 0x5567eeee41d0 2021-11-20T07:38:33.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48014/0 (socket says 172.21.15.71:48014) 2021-11-20T07:38:33.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 -- 172.21.15.71:0/4187468160 learned_addr learned my addr 172.21.15.71:0/4187468160 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84e59c700 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef082950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 -- 172.21.15.71:0/4187468160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 msgr2=0x5567ef082950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef082950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 -- 172.21.15.71:0/4187468160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567ef071e80 con 0x5567ef076fe0 2021-11-20T07:38:33.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fd838074f20 tx=0x7fd83804f040).ready entity=mon.0 client_cookie=4a5459cea657f408 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.185 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd844ff9700 1 -- 172.21.15.71:0/4187468160 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83804bf50 con 0x5567ef076fe0 2021-11-20T07:38:33.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd844ff9700 1 -- 172.21.15.71:0/4187468160 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd83804c530 con 0x5567ef076fe0 2021-11-20T07:38:33.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.163+0000 7fd844ff9700 1 -- 172.21.15.71:0/4187468160 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83804cd70 con 0x5567ef076fe0 2021-11-20T07:38:33.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4187468160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 msgr2=0x5567ef082e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.186 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fd838074f20 tx=0x7fd83804f040).stop 2021-11-20T07:38:33.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4187468160 shutdown_connections 2021-11-20T07:38:33.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef082e40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/4187468160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef082950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.187 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4187468160 >> 172.21.15.71:0/4187468160 conn(0x5567ef03c9b0 msgr2=0x5567ef0738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4187468160 shutdown_connections 2021-11-20T07:38:33.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- 172.21.15.71:0/4187468160 wait complete. 2021-11-20T07:38:33.188 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 -- start start 2021-11-20T07:38:33.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.164+0000 7fd8669ffb80 1 Processor -- start 2021-11-20T07:38:33.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef0d8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.189 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567ef0d9640 con 0x5567ef076fe0 2021-11-20T07:38:33.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567ef0d9780 con 0x5567eeee41d0 2021-11-20T07:38:33.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 10 client.0 ll_register_callbacks cb 0x5567ef036700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T07:38:33.190 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48020/0 (socket says 172.21.15.71:48020) 2021-11-20T07:38:33.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 -- 172.21.15.71:0/1898725848 learned_addr learned my addr 172.21.15.71:0/1898725848 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84e59c700 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef0d8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 -- 172.21.15.71:0/1898725848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 msgr2=0x5567ef0d8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef0d8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 -- 172.21.15.71:0/1898725848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567eefb8630 con 0x5567ef076fe0 2021-11-20T07:38:33.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd84dd9b700 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fd83804c290 tx=0x7fd83800b040).ready entity=mon.0 client_cookie=cbd19e672ea9e054 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd82d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2021-11-20T07:38:33.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd838014c20 con 0x5567ef076fe0 2021-11-20T07:38:33.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd838014d80 con 0x5567ef076fe0 2021-11-20T07:38:33.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.165+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5567ef071220 con 0x5567ef076fe0 2021-11-20T07:38:33.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8380748c0 con 0x5567ef076fe0 2021-11-20T07:38:33.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 10 client.14506 fetch_fsmap learned FSMap version 12 2021-11-20T07:38:33.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x7fd8501075b0 con 0x5567ef076fe0 2021-11-20T07:38:33.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd838021030 con 0x5567ef076fe0 2021-11-20T07:38:33.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 12) v1 ==== 1736+0+0 (secure 0 0 0) 0x7fd83805f040 con 0x5567ef076fe0 2021-11-20T07:38:33.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 10 client.14506 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T07:38:33.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 10 client.14506 resolve_mds: resolved name 'cephfs.smithi071.xtyomb' to daemon mds.cephfs.smithi071.xtyomb 2021-11-20T07:38:33.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 20 client.14506 populate_metadata read hostname 'smithi071' 2021-11-20T07:38:33.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef0def20 0x5567ef0e1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 4 client.14506 mds_command: new command op to 14430 tid=0[{"prefix": "status"}] 2021-11-20T07:38:33.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 --> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5567eefb70c0 con 0x5567ef0def20 2021-11-20T07:38:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd84e59c700 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef0def20 0x5567ef0e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd82d7fa700 10 client.14506 ms_handle_connect on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.166+0000 7fd84e59c700 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef0def20 0x5567ef0e1310 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=be449a251ab88fb3 server_cookie=b04fb59140f6b29a in_seq=0 out_seq=0 2021-11-20T07:38:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd82d7fa700 1 -- 172.21.15.71:0/1898725848 <== mds.0 v2:172.21.15.71:6826/657608204 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5567eefb70c0 con 0x5567ef0def20 2021-11-20T07:38:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd82d7fa700 10 client.14506 handle_command_reply: tid=0 2021-11-20T07:38:33.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 client.14506 shutdown 2021-11-20T07:38:33.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 msgr2=0x5567ef0d9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fd83804c290 tx=0x7fd83800b040).stop 2021-11-20T07:38:33.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 shutdown_connections 2021-11-20T07:38:33.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x5567ef0def20 0x5567ef0e1310 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5567eeee41d0 0x5567ef0d8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd82d7fa700 0 client.14506 ms_handle_reset on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:33.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 --2- 172.21.15.71:0/1898725848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567ef076fe0 0x5567ef0d9090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 >> 172.21.15.71:0/1898725848 conn(0x5567ef03c9b0 msgr2=0x5567ef082950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 shutdown_connections 2021-11-20T07:38:33.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 1 -- 172.21.15.71:0/1898725848 wait complete. 2021-11-20T07:38:33.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.167+0000 7fd8669ffb80 20 client.14506 trim_cache size 0 max 16384 2021-11-20T07:38:33.200 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2021-11-20T07:38:33.200 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "16910ea2-49d4-11ec-8c2c-001a4aab830c", 2021-11-20T07:38:33.200 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 0, 2021-11-20T07:38:33.201 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 14430, 2021-11-20T07:38:33.201 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2021-11-20T07:38:33.201 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2021-11-20T07:38:33.201 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2021-11-20T07:38:33.201 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 14.730402685, 2021-11-20T07:38:33.202 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 8, 2021-11-20T07:38:33.202 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 41, 2021-11-20T07:38:33.202 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 41, 2021-11-20T07:38:33.202 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 15.730272402000001 2021-11-20T07:38:33.203 INFO:teuthology.orchestra.run.smithi071.stdout:} 2021-11-20T07:38:33.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 msgr2=0x7fd83c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 0x7fd83c05e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd84800b620 tx=0x7fd848004d10).stop 2021-11-20T07:38:33.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 msgr2=0x7fd85010bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fd8380026d0 tx=0x7fd838009de0).stop 2021-11-20T07:38:33.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 shutdown_connections 2021-11-20T07:38:33.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8500f5590 0x7fd8500604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd83c05c2a0 0x7fd83c05e760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 --2- 172.21.15.71:0/17632180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8500f6830 0x7fd85010bd80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 >> 172.21.15.71:0/17632180 conn(0x7fd8500f0d10 msgr2=0x7fd8500fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 shutdown_connections 2021-11-20T07:38:33.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.168+0000 7fd854e54700 1 -- 172.21.15.71:0/17632180 wait complete. 2021-11-20T07:38:33.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi112.bxcrgi status 2021-11-20T07:38:33.326 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/1283567383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 msgr2=0x7f9ea40f7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.326 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/1283567383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea40f7060 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f9e94004660 tx=0x7f9e94009b30).stop 2021-11-20T07:38:33.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/1283567383 shutdown_connections 2021-11-20T07:38:33.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/1283567383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 0x7f9ea40f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/1283567383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea40f7060 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/1283567383 >> 172.21.15.71:0/1283567383 conn(0x7f9ea40f04a0 msgr2=0x7f9ea40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.327 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/1283567383 shutdown_connections 2021-11-20T07:38:33.328 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.322+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/1283567383 wait complete. 2021-11-20T07:38:33.328 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 Processor -- start 2021-11-20T07:38:33.328 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 -- start start 2021-11-20T07:38:33.328 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 0x7f9ea40f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea410c0a0 con 0x7f9ea40f6c40 2021-11-20T07:38:33.329 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea9f9a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea410c1e0 con 0x7f9ea40f5950 2021-11-20T07:38:33.330 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.330 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48030/0 (socket says 172.21.15.71:48030) 2021-11-20T07:38:33.330 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 -- 172.21.15.71:0/3893040237 learned_addr learned my addr 172.21.15.71:0/3893040237 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.330 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 0x7f9ea40f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 -- 172.21.15.71:0/3893040237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 msgr2=0x7f9ea40f52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 0x7f9ea40f52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 -- 172.21.15.71:0/3893040237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e94005040 con 0x7f9ea40f6c40 2021-11-20T07:38:33.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9e8c002700 tx=0x7f9e8c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.331 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.323+0000 7f9ea0ff9700 1 -- 172.21.15.71:0/3893040237 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c009320 con 0x7f9ea40f6c40 2021-11-20T07:38:33.332 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.324+0000 7f9ea0ff9700 1 -- 172.21.15.71:0/3893040237 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e8c00f040 con 0x7f9ea40f6c40 2021-11-20T07:38:33.332 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.324+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea410c440 con 0x7f9ea40f6c40 2021-11-20T07:38:33.332 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.324+0000 7f9ea0ff9700 1 -- 172.21.15.71:0/3893040237 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c014870 con 0x7f9ea40f6c40 2021-11-20T07:38:33.332 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.324+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea410c9d0 con 0x7f9ea40f6c40 2021-11-20T07:38:33.333 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.325+0000 7f9ea0ff9700 1 -- 172.21.15.71:0/3893040237 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e8c009480 con 0x7f9ea40f6c40 2021-11-20T07:38:33.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.325+0000 7f9ea0ff9700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 0x7f9e9005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.325+0000 7f9ea0ff9700 1 -- 172.21.15.71:0/3893040237 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9e8c076150 con 0x7f9ea40f6c40 2021-11-20T07:38:33.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.325+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 0x7f9e9005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.325+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 0x7f9e9005e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9e94009f50 tx=0x7f9e94009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 Processor -- start 2021-11-20T07:38:33.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 -- start start 2021-11-20T07:38:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f5fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81e9d4d0 con 0x558b81dcc1d0 2021-11-20T07:38:33.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81eb13e0 con 0x558b81f5efe0 2021-11-20T07:38:33.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.327+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48034/0 (socket says 172.21.15.71:48034) 2021-11-20T07:38:33.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9ea37fe700 1 -- 172.21.15.71:0/3237815424 learned_addr learned my addr 172.21.15.71:0/3237815424 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f5fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9ea37fe700 1 -- 172.21.15.71:0/3237815424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 msgr2=0x558b81f5fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f5fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9ea37fe700 1 -- 172.21.15.71:0/3237815424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b81f2d3c0 con 0x558b81dcc1d0 2021-11-20T07:38:33.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f5fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:38:33.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9e94011000 tx=0x7f9e9401d000).ready entity=mon.0 client_cookie=c75d661c6c4376ba server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9e997fa700 1 -- 172.21.15.71:0/3237815424 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9400e040 con 0x558b81dcc1d0 2021-11-20T07:38:33.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9e997fa700 1 -- 172.21.15.71:0/3237815424 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e94019040 con 0x558b81dcc1d0 2021-11-20T07:38:33.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.328+0000 7f9e997fa700 1 -- 172.21.15.71:0/3237815424 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9401dac0 con 0x558b81dcc1d0 2021-11-20T07:38:33.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/3237815424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 msgr2=0x558b81f5eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9e94011000 tx=0x7f9e9401d000).stop 2021-11-20T07:38:33.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/3237815424 shutdown_connections 2021-11-20T07:38:33.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f5fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/3237815424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f5eaf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/3237815424 >> 172.21.15.71:0/3237815424 conn(0x558b81f249b0 msgr2=0x558b81f59fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/3237815424 shutdown_connections 2021-11-20T07:38:33.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/3237815424 wait complete. 2021-11-20T07:38:33.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 -- start start 2021-11-20T07:38:33.347 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.329+0000 7f9ebbb45b80 1 Processor -- start 2021-11-20T07:38:33.347 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.347 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f67b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.347 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81eb1520 con 0x558b81dcc1d0 2021-11-20T07:38:33.347 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81eb1aa0 con 0x558b81f5efe0 2021-11-20T07:38:33.348 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 10 client.0 ll_register_callbacks cb 0x558b81f1e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T07:38:33.348 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.348 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48038/0 (socket says 172.21.15.71:48038) 2021-11-20T07:38:33.348 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 -- 172.21.15.71:0/2894613777 learned_addr learned my addr 172.21.15.71:0/2894613777 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.349 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f67b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.349 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 -- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 msgr2=0x558b81f67b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.349 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f67b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.349 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 -- 172.21.15.71:0/2894613777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea40f6480 con 0x558b81dcc1d0 2021-11-20T07:38:33.350 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f67b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:38:33.350 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9e940041e0 tx=0x7f9e9401d000).ready entity=mon.0 client_cookie=84606ca29b78e1cd server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.350 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e827fc700 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2021-11-20T07:38:33.351 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9400e040 con 0x558b81dcc1d0 2021-11-20T07:38:33.351 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9e9401dc90 con 0x558b81dcc1d0 2021-11-20T07:38:33.351 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b81f2cce0 con 0x558b81dcc1d0 2021-11-20T07:38:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e94007950 con 0x558b81dcc1d0 2021-11-20T07:38:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 10 client.14514 fetch_fsmap learned FSMap version 12 2021-11-20T07:38:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.330+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x558b81ea0630 con 0x558b81dcc1d0 2021-11-20T07:38:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x558b81f2cce0 con 0x558b81dcc1d0 2021-11-20T07:38:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 12) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f9e9401f740 con 0x558b81dcc1d0 2021-11-20T07:38:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 10 client.14514 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T07:38:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 10 client.14514 resolve_mds: resolved name 'cephfs.smithi112.bxcrgi' to daemon mds.cephfs.smithi112.bxcrgi 2021-11-20T07:38:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 20 client.14514 populate_metadata read hostname 'smithi071' 2021-11-20T07:38:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81f70310 0x558b81f72700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 4 client.14514 mds_command: new command op to 24253 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T07:38:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 --> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b81ea7df0 con 0x558b81f70310 2021-11-20T07:38:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81f70310 0x558b81f72700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9e827fc700 10 client.14514 ms_handle_connect on v2:172.21.15.112:6824/3641386819 2021-11-20T07:38:33.355 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.331+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81f70310 0x558b81f72700 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=52635e9d2a250b7d server_cookie=77286b43a0944e7b in_seq=0 out_seq=0 2021-11-20T07:38:33.355 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.332+0000 7f9e827fc700 1 -- 172.21.15.71:0/2894613777 <== mds.1 v2:172.21.15.112:6824/3641386819 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x558b81ea7df0 con 0x558b81f70310 2021-11-20T07:38:33.355 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.332+0000 7f9e827fc700 10 client.14514 handle_command_reply: tid=0 2021-11-20T07:38:33.355 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.332+0000 7f9ebbb45b80 1 client.14514 shutdown 2021-11-20T07:38:33.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 msgr2=0x558b81f67680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9e940041e0 tx=0x7f9e9401d000).stop 2021-11-20T07:38:33.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 shutdown_connections 2021-11-20T07:38:33.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81f70310 0x558b81f72700 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81f67680 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9e827fc700 0 client.14514 ms_handle_reset on v2:172.21.15.112:6824/3641386819 2021-11-20T07:38:33.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2894613777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81f67b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 >> 172.21.15.71:0/2894613777 conn(0x558b81f249b0 msgr2=0x558b81f5b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 shutdown_connections 2021-11-20T07:38:33.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2894613777 wait complete. 2021-11-20T07:38:33.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.342+0000 7f9ebbb45b80 20 client.14514 trim_cache size 0 max 16384 2021-11-20T07:38:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.358+0000 7f9ebbb45b80 1 Processor -- start 2021-11-20T07:38:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.358+0000 7f9ebbb45b80 1 -- start start 2021-11-20T07:38:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81dcc1d0 0x558b81f5f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81e7f4a0 con 0x558b81f5efe0 2021-11-20T07:38:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81ea7df0 con 0x558b81dcc1d0 2021-11-20T07:38:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48046/0 (socket says 172.21.15.71:48046) 2021-11-20T07:38:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 -- 172.21.15.71:0/2815415081 learned_addr learned my addr 172.21.15.71:0/2815415081 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81dcc1d0 0x558b81f5f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 -- 172.21.15.71:0/2815415081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81dcc1d0 msgr2=0x558b81f5f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81dcc1d0 0x558b81f5f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 -- 172.21.15.71:0/2815415081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b81f5b2f0 con 0x558b81f5efe0 2021-11-20T07:38:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.359+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9e8c009bd0 tx=0x7f9e8c04c710).ready entity=mon.0 client_cookie=a9f978604ab8a74a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9e99ffb700 1 -- 172.21.15.71:0/2815415081 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c04ac10 con 0x558b81f5efe0 2021-11-20T07:38:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9e99ffb700 1 -- 172.21.15.71:0/2815415081 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e8c074660 con 0x558b81f5efe0 2021-11-20T07:38:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9e99ffb700 1 -- 172.21.15.71:0/2815415081 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c0077b0 con 0x558b81f5efe0 2021-11-20T07:38:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2815415081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 msgr2=0x558b81f5fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9e8c009bd0 tx=0x7f9e8c04c710).stop 2021-11-20T07:38:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2815415081 shutdown_connections 2021-11-20T07:38:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81f5efe0 0x558b81f5fae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/2815415081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81dcc1d0 0x558b81f5f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2815415081 >> 172.21.15.71:0/2815415081 conn(0x558b81f249b0 msgr2=0x558b81f579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2815415081 shutdown_connections 2021-11-20T07:38:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.360+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/2815415081 wait complete. 2021-11-20T07:38:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 -- start start 2021-11-20T07:38:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 Processor -- start 2021-11-20T07:38:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81fbce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81fbd3c0 con 0x558b81dcc1d0 2021-11-20T07:38:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b81fbd500 con 0x558b81f5efe0 2021-11-20T07:38:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ebbb45b80 10 client.0 ll_register_callbacks cb 0x558b81f1e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T07:38:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48048/0 (socket says 172.21.15.71:48048) 2021-11-20T07:38:33.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 -- 172.21.15.71:0/1746518521 learned_addr learned my addr 172.21.15.71:0/1746518521 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:33.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81fbce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 -- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 msgr2=0x558b81fbce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:33.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81fbce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 -- 172.21.15.71:0/1746518521 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea410c9d0 con 0x558b81dcc1d0 2021-11-20T07:38:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9ea37fe700 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f9e9401d630 tx=0x7f9e9401d000).ready entity=mon.0 client_cookie=13e32b187eb854a2 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9e827fc700 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2021-11-20T07:38:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.361+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9400e040 con 0x558b81dcc1d0 2021-11-20T07:38:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9e94029670 con 0x558b81dcc1d0 2021-11-20T07:38:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b81f59220 con 0x558b81dcc1d0 2021-11-20T07:38:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e94029830 con 0x558b81dcc1d0 2021-11-20T07:38:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 10 client.14518 fetch_fsmap learned FSMap version 12 2021-11-20T07:38:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x7f9ea410f8f0 con 0x558b81dcc1d0 2021-11-20T07:38:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x558b81f59220 con 0x558b81dcc1d0 2021-11-20T07:38:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 12) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f9e94031bd0 con 0x558b81dcc1d0 2021-11-20T07:38:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 10 client.14518 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T07:38:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 10 client.14518 resolve_mds: resolved name 'cephfs.smithi112.bxcrgi' to daemon mds.cephfs.smithi112.bxcrgi 2021-11-20T07:38:33.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 20 client.14518 populate_metadata read hostname 'smithi071' 2021-11-20T07:38:33.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81fc2ca0 0x558b81fc5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:33.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 4 client.14518 mds_command: new command op to 24253 tid=0[{"prefix": "status"}] 2021-11-20T07:38:33.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 --> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558b81e9f0c0 con 0x558b81fc2ca0 2021-11-20T07:38:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81fc2ca0 0x558b81fc5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e827fc700 10 client.14518 ms_handle_connect on v2:172.21.15.112:6824/3641386819 2021-11-20T07:38:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.362+0000 7f9e9bfff700 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81fc2ca0 0x558b81fc5090 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=a2fae2a8fe84b438 server_cookie=f26943756ff66b41 in_seq=0 out_seq=0 2021-11-20T07:38:33.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9e827fc700 1 -- 172.21.15.71:0/1746518521 <== mds.1 v2:172.21.15.112:6824/3641386819 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558b81e9f0c0 con 0x558b81fc2ca0 2021-11-20T07:38:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9e827fc700 10 client.14518 handle_command_reply: tid=0 2021-11-20T07:38:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 client.14518 shutdown 2021-11-20T07:38:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 msgr2=0x558b81fbc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f9e9401d630 tx=0x7f9e9401d000).stop 2021-11-20T07:38:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 shutdown_connections 2021-11-20T07:38:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:6824/3641386819,v1:172.21.15.112:6825/3641386819] conn(0x558b81fc2ca0 0x558b81fc5090 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558b81dcc1d0 0x558b81fbc920 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 --2- 172.21.15.71:0/1746518521 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b81f5efe0 0x558b81fbce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9e827fc700 0 client.14518 ms_handle_reset on v2:172.21.15.112:6824/3641386819 2021-11-20T07:38:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 >> 172.21.15.71:0/1746518521 conn(0x558b81f249b0 msgr2=0x558b81f66dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 shutdown_connections 2021-11-20T07:38:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 1 -- 172.21.15.71:0/1746518521 wait complete. 2021-11-20T07:38:33.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ebbb45b80 20 client.14518 trim_cache size 0 max 16384 2021-11-20T07:38:33.384 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2021-11-20T07:38:33.385 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "16910ea2-49d4-11ec-8c2c-001a4aab830c", 2021-11-20T07:38:33.385 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 1, 2021-11-20T07:38:33.386 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 24253, 2021-11-20T07:38:33.386 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2021-11-20T07:38:33.387 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2021-11-20T07:38:33.387 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2021-11-20T07:38:33.387 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 13.921937526000001, 2021-11-20T07:38:33.387 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 8, 2021-11-20T07:38:33.388 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 41, 2021-11-20T07:38:33.388 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 41, 2021-11-20T07:38:33.388 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 18.519692260999999 2021-11-20T07:38:33.388 INFO:teuthology.orchestra.run.smithi071.stdout:} 2021-11-20T07:38:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 msgr2=0x7f9e9005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 0x7f9e9005e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9e94009f50 tx=0x7f9e94009b30).stop 2021-11-20T07:38:33.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 msgr2=0x7f9ea410bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:33.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.363+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9e8c002700 tx=0x7f9e8c009e00).stop 2021-11-20T07:38:33.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 shutdown_connections 2021-11-20T07:38:33.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ea40f5950 0x7f9ea40f52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9e9005c2a0 0x7f9e9005e760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 --2- 172.21.15.71:0/3893040237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9ea40f6c40 0x7f9ea410bbb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 >> 172.21.15.71:0/3893040237 conn(0x7f9ea40f04a0 msgr2=0x7f9ea40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 shutdown_connections 2021-11-20T07:38:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:33.364+0000 7f9ea9f9a700 1 -- 172.21.15.71:0/3893040237 wait complete. 2021-11-20T07:38:33.393 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2021-11-20T07:38:33.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.394 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p /home/ubuntu/cephtest/mnt.0 2021-11-20T07:38:33.410 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-11-20T07:38:33.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T07:38:33.514 INFO:teuthology.orchestra.run.smithi071.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T07:38:33.514 INFO:teuthology.orchestra.run.smithi071.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T07:38:33.628 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: cluster 2021-11-20T07:38:32.365083 2021-11-20T07:38:33.628 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2021-11-20T07:38:33.628 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: audit 2021-11-20T07:38:32.671572+0000 mon.smithi071 ( 2021-11-20T07:38:33.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: mon.0) 528 : audit [DBG] from='client.? 172.21.15.71:0/1989499498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:33.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: audit 2021-11-20T07:38:33 2021-11-20T07:38:33.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: .006042+0000 mon.smithi071 (mon.0) 529 : audit 2021-11-20T07:38:33.629 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:33 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/584742627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:33.652 INFO:teuthology.orchestra.run.smithi071.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T07:38:33.653 INFO:teuthology.orchestra.run.smithi071.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T07:38:33.656 INFO:tasks.cephfs.kernel_mount:mount command passed 2021-11-20T07:38:33.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2021-11-20T07:38:33.695 INFO:tasks.kclient:client.1 config is {} 2021-11-20T07:38:33.695 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T07:38:33.695 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi112.front.sepia.ceph.com 2021-11-20T07:38:33.696 INFO:tasks.cephfs.mount:self.client.name = client.1 2021-11-20T07:38:33.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2021-11-20T07:38:33.696 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T07:38:33.697 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T07:38:33.699 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T07:38:33.699 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2021-11-20T07:38:33.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.700 DEBUG:teuthology.orchestra.run.smithi112:> ip addr 2021-11-20T07:38:33.721 INFO:teuthology.orchestra.run.smithi112.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T07:38:33.721 INFO:teuthology.orchestra.run.smithi112.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T07:38:33.721 INFO:teuthology.orchestra.run.smithi112.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T07:38:33.721 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:33.722 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 ::1/128 scope host 2021-11-20T07:38:33.722 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:33.722 INFO:teuthology.orchestra.run.smithi112.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:33.723 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2c brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:33.723 INFO:teuthology.orchestra.run.smithi112.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:33.723 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2d brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:33.723 INFO:teuthology.orchestra.run.smithi112.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T07:38:33.723 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:68 brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:33.724 INFO:teuthology.orchestra.run.smithi112.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T07:38:33.725 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:69 brd ff:ff:ff:ff:ff:ff 2021-11-20T07:38:33.725 INFO:teuthology.orchestra.run.smithi112.stdout: inet 172.21.15.112/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T07:38:33.725 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft 42258sec preferred_lft 42258sec 2021-11-20T07:38:33.725 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 fe80::ec4:7aff:fe8f:cb69/64 scope link noprefixroute 2021-11-20T07:38:33.726 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2021-11-20T07:38:33.726 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T07:38:33.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.727 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2021-11-20T07:38:33.727 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2021-11-20T07:38:33.728 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add name ceph-brx type bridge 2021-11-20T07:38:33.728 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr flush dev ceph-brx 2021-11-20T07:38:33.728 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set ceph-brx up 2021-11-20T07:38:33.729 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T07:38:33.729 DEBUG:teuthology.orchestra.run.smithi112:> ') 2021-11-20T07:38:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:33 smithi112 conmon[36062]: cluster 2021-11-20T07:38:32.365083+0000 mgr.smithi071.ynqmwn (mgr.14162) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2021-11-20T07:38:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:33 smithi112 conmon[36062]: audit 2021-11-20T07:38:32 2021-11-20T07:38:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:33 smithi112 conmon[36062]: .671572+0000 mon.smithi071 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.71:0/1989499498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:33 smithi112 conmon[36062]: audit 2021-11-20T07:38:33.006042+0000 mon.smithi071 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.71:0/584742627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:33.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.899 DEBUG:teuthology.orchestra.run.smithi112:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T07:38:33.965 INFO:teuthology.orchestra.run.smithi112.stdout:1 2021-11-20T07:38:33.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:33.966 DEBUG:teuthology.orchestra.run.smithi112:> route 2021-11-20T07:38:34.033 INFO:teuthology.orchestra.run.smithi112.stdout:Kernel IP routing table 2021-11-20T07:38:34.033 INFO:teuthology.orchestra.run.smithi112.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T07:38:34.033 INFO:teuthology.orchestra.run.smithi112.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T07:38:34.034 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T07:38:34.034 INFO:teuthology.orchestra.run.smithi112.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T07:38:34.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.035 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2021-11-20T07:38:34.035 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2021-11-20T07:38:34.036 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T07:38:34.036 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T07:38:34.036 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T07:38:34.037 DEBUG:teuthology.orchestra.run.smithi112:> ') 2021-11-20T07:38:34.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.320 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2021-11-20T07:38:34.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.337 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list-id 2021-11-20T07:38:34.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.393 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2021-11-20T07:38:34.393 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2021-11-20T07:38:34.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2021-11-20T07:38:34.394 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2021-11-20T07:38:34.394 DEBUG:teuthology.orchestra.run.smithi112:> ') 2021-11-20T07:38:34.484 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2021-11-20T07:38:34.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.485 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2021-11-20T07:38:34.485 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2021-11-20T07:38:34.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2021-11-20T07:38:34.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T07:38:34.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2021-11-20T07:38:34.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2021-11-20T07:38:34.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2021-11-20T07:38:34.487 DEBUG:teuthology.orchestra.run.smithi112:> ') 2021-11-20T07:38:34.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T07:38:34.718 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2021-11-20T07:38:34.718 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2021-11-20T07:38:34.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set brx.0 up 2021-11-20T07:38:34.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T07:38:34.719 DEBUG:teuthology.orchestra.run.smithi112:> ') 2021-11-20T07:38:34.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T07:38:34.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- 172.21.15.71:0/3275709834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 msgr2=0x7f75e00f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:34.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 --2- 172.21.15.71:0/3275709834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e00f6a90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f75d4004660 tx=0x7f75d4009b30).stop 2021-11-20T07:38:34.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- 172.21.15.71:0/3275709834 shutdown_connections 2021-11-20T07:38:34.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 --2- 172.21.15.71:0/3275709834 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 0x7f75e00f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:34.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 --2- 172.21.15.71:0/3275709834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e00f6a90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:34.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- 172.21.15.71:0/3275709834 >> 172.21.15.71:0/3275709834 conn(0x7f75e00f1db0 msgr2=0x7f75e00f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:34.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- 172.21.15.71:0/3275709834 shutdown_connections 2021-11-20T07:38:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- 172.21.15.71:0/3275709834 wait complete. 2021-11-20T07:38:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 Processor -- start 2021-11-20T07:38:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.952+0000 7f75e7250700 1 -- start start 2021-11-20T07:38:34.958 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:34.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 0x7f75e01073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:34.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0107960 con 0x7f75e00f6670 2021-11-20T07:38:34.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0107aa0 con 0x7f75e00f7910 2021-11-20T07:38:34.959 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48064/0 (socket says 172.21.15.71:48064) 2021-11-20T07:38:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 -- 172.21.15.71:0/2069349217 learned_addr learned my addr 172.21.15.71:0/2069349217 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75dffff700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 0x7f75e01073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 -- 172.21.15.71:0/2069349217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 msgr2=0x7f75e01073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 0x7f75e01073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:34.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 -- 172.21.15.71:0/2069349217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4005040 con 0x7f75e00f6670 2021-11-20T07:38:34.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e4fec700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f75d4009f50 tx=0x7f75d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:34.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d4018070 con 0x7f75e00f6670 2021-11-20T07:38:34.961 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75d4013410 con 0x7f75e00f6670 2021-11-20T07:38:34.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d401c3c0 con 0x7f75e00f6670 2021-11-20T07:38:34.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e010c4e0 con 0x7f75e00f6670 2021-11-20T07:38:34.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.953+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e010caa0 con 0x7f75e00f6670 2021-11-20T07:38:34.962 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.954+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e0027a00 con 0x7f75e00f6670 2021-11-20T07:38:34.963 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.955+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f75d401d420 con 0x7f75e00f6670 2021-11-20T07:38:34.964 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.955+0000 7f75dd7fa700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 0x7f75d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:34.964 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.955+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f75d4077c20 con 0x7f75e00f6670 2021-11-20T07:38:34.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.955+0000 7f75dffff700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 0x7f75d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:34.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.955+0000 7f75dffff700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 0x7f75d005e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f75cc0041d0 tx=0x7f75cc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:34.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:34.959+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75d404a8e0 con 0x7f75e00f6670 2021-11-20T07:38:35.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.106+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75e00fee00 con 0x7f75e00f6670 2021-11-20T07:38:35.109 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.107+0000 7f75dd7fa700 1 -- 172.21.15.71:0/2069349217 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14034 (secure 0 0 0) 0x7f75d404df00 con 0x7f75e00f6670 2021-11-20T07:38:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:35.109 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":41,"fsid":"16910ea2-49d4-11ec-8c2c-001a4aab830c","created":"2021-11-20T07:33:21.630567+0000","modified":"2021-11-20T07:38:14.416035+0000","last_up_change":"2021-11-20T07:37:15.899596+0000","last_in_change":"2021-11-20T07:36:59.847913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T07:36:35.321439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T07:38:11.027597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T07:38:11.644741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06586a57-2910-4cab-8442-30fd5e92b95d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6803","nonce":1275932233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6805","nonce":1275932233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6809","nonce":1275932233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":1275932233},{"type":"v1","addr":"172.21.15.71:6807","nonce":1275932233}]},"public_addr":"172.21.15.71:6803/1275932233","cluster_addr":"172.21.15.71:6805/1275932233","heartbeat_back_addr":"172.21.15.71:6809/1275932233","heartbeat_front_addr":"172.21.15.71:6807/1275932233","state":["exists","up"]},{"osd":1,"uuid":"32e5e3ee-bd5a-4777-94b2-d475e3515783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6811","nonce":12679585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6813","nonce":12679585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6817","nonce":12679585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":12679585},{"type":"v1","addr":"172.21.15.71:6815","nonce":12679585}]},"public_addr":"172.21.15.71:6811/12679585","cluster_addr":"172.21.15.71:6813/12679585","heartbeat_back_addr":"172.21.15.71:6817/12679585","heartbeat_front_addr":"172.21.15.71:6815/12679585","state":["exists","up"]},{"osd":2,"uuid":"c011d0ec-7e59-44fb-bc53-eb3324d914bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6819","nonce":3317104048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6821","nonce":3317104048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6825","nonce":3317104048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":3317104048},{"type":"v1","addr":"172.21.15.71:6823","nonce":3317104048}]},"public_addr":"172.21.15.71:6819/3317104048","cluster_addr":"172.21.15.71:6821/3317104048","heartbeat_back_addr":"172.21.15.71:6825/3317104048","heartbeat_front_addr":"172.21.15.71:6823/3317104048","state":["exists","up"]},{"osd":3,"uuid":"5fe79473-cc34-4424-876a-09b500aae5a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6801","nonce":3567947244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6803","nonce":3567947244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6807","nonce":3567947244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3567947244},{"type":"v1","addr":"172.21.15.112:6805","nonce":3567947244}]},"public_addr":"172.21.15.112:6801/3567947244","cluster_addr":"172.21.15.112:6803/3567947244","heartbeat_back_addr":"172.21.15.112:6807/3567947244","heartbeat_front_addr":"172.21.15.112:6805/3567947244","state":["exists","up"]},{"osd":4,"uuid":"438e6bcc-10d6-473a-8732-7e2f399906d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":297260},{"type":"v1","addr":"172.21.15.112:6809","nonce":297260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":297260},{"type":"v1","addr":"172.21.15.112:6811","nonce":297260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":297260},{"type":"v1","addr":"172.21.15.112:6815","nonce":297260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":297260},{"type":"v1","addr":"172.21.15.112:6813","nonce":297260}]},"public_addr":"172.21.15.112:6809/297260","cluster_addr":"172.21.15.112:6811/297260","heartbeat_back_addr":"172.21.15.112:6815/297260","heartbeat_front_addr":"172.21.15.112:6813/297260","state":["exists","up"]},{"osd":5,"uuid":"cfbcc628-860f-4297-85e1-58064adc31c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6817","nonce":3514154095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6819","nonce":3514154095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6823","nonce":3514154095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3514154095},{"type":"v1","addr":"172.21.15.112:6821","nonce":3514154095}]},"public_addr":"172.21.15.112:6817/3514154095","cluster_addr":"172.21.15.112:6819/3514154095","heartbeat_back_addr":"172.21.15.112:6823/3514154095","heartbeat_front_addr":"172.21.15.112:6821/3514154095","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:35:48.763754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:06.560028+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:24.073475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:40.003790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:36:57.223732+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T07:37:14.145361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6801/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6800/4211885430":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/969021789":"2021-11-21T07:34:16.294071+0000","172.21.15.71:0/1610295319":"2021-11-21T07:34:16.294071+0000","172.21.15.71:6801/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:6800/2583959496":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/730899033":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/1568873093":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6800/896254714":"2021-11-21T07:33:39.857148+0000","172.21.15.71:0/578274960":"2021-11-21T07:34:02.812128+0000","172.21.15.71:0/3866584008":"2021-11-21T07:33:39.857148+0000","172.21.15.71:6801/896254714":"2021-11-21T07:33:39.857148+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T07:38:35.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 msgr2=0x7f75d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 0x7f75d005e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f75cc0041d0 tx=0x7f75cc009040).stop 2021-11-20T07:38:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 msgr2=0x7f75e0106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f75d4009f50 tx=0x7f75d4009b30).stop 2021-11-20T07:38:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 shutdown_connections 2021-11-20T07:38:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75d005c2a0 0x7f75d005e760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75e00f6670 0x7f75e0106ec0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 --2- 172.21.15.71:0/2069349217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75e00f7910 0x7f75e01073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 >> 172.21.15.71:0/2069349217 conn(0x7f75e00f1db0 msgr2=0x7f75e00fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 shutdown_connections 2021-11-20T07:38:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.109+0000 7f75e7250700 1 -- 172.21.15.71:0/2069349217 wait complete. 2021-11-20T07:38:35.122 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T07:36:35.321439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T07:38:11.027597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T07:38:11.644741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T07:38:35.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T07:38:35.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 -- 172.21.15.71:0/1995960671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f55b0 msgr2=0x7f97d80f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 --2- 172.21.15.71:0/1995960671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f55b0 0x7f97d80f59d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f97c8004660 tx=0x7f97c8009b30).stop 2021-11-20T07:38:35.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 -- 172.21.15.71:0/1995960671 shutdown_connections 2021-11-20T07:38:35.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 --2- 172.21.15.71:0/1995960671 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f6850 0x7f97d80f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 --2- 172.21.15.71:0/1995960671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f55b0 0x7f97d80f59d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 -- 172.21.15.71:0/1995960671 >> 172.21.15.71:0/1995960671 conn(0x7f97d80f0d10 msgr2=0x7f97d80f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 -- 172.21.15.71:0/1995960671 shutdown_connections 2021-11-20T07:38:35.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.240+0000 7f97ddc66700 1 -- 172.21.15.71:0/1995960671 wait complete. 2021-11-20T07:38:35.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 Processor -- start 2021-11-20T07:38:35.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 -- start start 2021-11-20T07:38:35.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 0x7f97d805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d810c0b0 con 0x7f97d80f6850 2021-11-20T07:38:35.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97ddc66700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d810c1f0 con 0x7f97d80f55b0 2021-11-20T07:38:35.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.241+0000 7f97cffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d77fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 0x7f97d805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97cffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48076/0 (socket says 172.21.15.71:48076) 2021-11-20T07:38:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97cffff700 1 -- 172.21.15.71:0/2933329057 learned_addr learned my addr 172.21.15.71:0/2933329057 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:35.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d77fe700 1 -- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 msgr2=0x7f97d810bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d77fe700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d77fe700 1 -- 172.21.15.71:0/2933329057 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8005040 con 0x7f97d80f55b0 2021-11-20T07:38:35.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97cffff700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T07:38:35.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d77fe700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 0x7f97d805ff20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f97c8000c00 tx=0x7f97c800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c8019070 con 0x7f97d80f55b0 2021-11-20T07:38:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.242+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97c8007e70 con 0x7f97d80f55b0 2021-11-20T07:38:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.243+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c8014c00 con 0x7f97d80f55b0 2021-11-20T07:38:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.243+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d810c3f0 con 0x7f97d80f55b0 2021-11-20T07:38:35.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.244+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d810c900 con 0x7f97d80f55b0 2021-11-20T07:38:35.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.245+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f97c8014420 con 0x7f97d80f55b0 2021-11-20T07:38:35.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.245+0000 7f97d4ff9700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 0x7f97c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.246+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f97c8076650 con 0x7f97d80f55b0 2021-11-20T07:38:35.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.246+0000 7f97cffff700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 0x7f97c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.246+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d8046330 con 0x7f97d80f55b0 2021-11-20T07:38:35.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.250+0000 7f97cffff700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 0x7f97c405e760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f97c0007a30 tx=0x7f97c000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.251+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97c8049280 con 0x7f97d80f55b0 2021-11-20T07:38:35.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:35 smithi071 conmon[33687]: audit 2021-11-20T07:38:34.359440+0000 mon.smithi071 (mon.0) 530 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:35.416 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:35 smithi071 conmon[33687]: audit 2021-11-20T07:38:34.362340+0000 mon.smithi071 (mon.0) 531 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:35.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:35 smithi071 conmon[33687]: cluster 2021-11-20T07:38:34.365421+0000 mgr.smithi071.ynqmwn (mgr.14162) 198 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2021-11-20T07:38:35.417 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:35 smithi071 conmon[33687]: audit 2021-11-20T07:38:35.108173+0000 mon.smithi071 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.71:0/2069349217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:35.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.417+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f97d8027a00 con 0x7f97d80f55b0 2021-11-20T07:38:35.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.418+0000 7f97d4ff9700 1 -- 172.21.15.71:0/2933329057 <== mon.1 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f97c804e020 con 0x7f97d80f55b0 2021-11-20T07:38:35.420 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2021-11-20T07:38:35.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 msgr2=0x7f97c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 0x7f97c405e760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f97c0007a30 tx=0x7f97c000b040).stop 2021-11-20T07:38:35.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 msgr2=0x7f97d805ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 0x7f97d805ff20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f97c8000c00 tx=0x7f97c800b040).stop 2021-11-20T07:38:35.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 shutdown_connections 2021-11-20T07:38:35.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97d80f55b0 0x7f97d805ff20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f97c405c2a0 0x7f97c405e760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 --2- 172.21.15.71:0/2933329057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97d80f6850 0x7f97d810bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 >> 172.21.15.71:0/2933329057 conn(0x7f97d80f0d10 msgr2=0x7f97d80fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 shutdown_connections 2021-11-20T07:38:35.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.420+0000 7f97ddc66700 1 -- 172.21.15.71:0/2933329057 wait complete. 2021-11-20T07:38:35.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T07:38:35.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 -- 172.21.15.71:0/3234312843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 msgr2=0x7fd9500f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 --2- 172.21.15.71:0/3234312843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd9500f5990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd94c004660 tx=0x7fd94c009b30).stop 2021-11-20T07:38:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 -- 172.21.15.71:0/3234312843 shutdown_connections 2021-11-20T07:38:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 --2- 172.21.15.71:0/3234312843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9500f6810 0x7fd9500f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 --2- 172.21.15.71:0/3234312843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd9500f5990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 -- 172.21.15.71:0/3234312843 >> 172.21.15.71:0/3234312843 conn(0x7fd9500f0cf0 msgr2=0x7fd9500f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 -- 172.21.15.71:0/3234312843 shutdown_connections 2021-11-20T07:38:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.556+0000 7fd957e4e700 1 -- 172.21.15.71:0/3234312843 wait complete. 2021-11-20T07:38:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 Processor -- start 2021-11-20T07:38:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 -- start start 2021-11-20T07:38:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9500f6810 0x7fd950107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950107940 con 0x7fd9500f5570 2021-11-20T07:38:35.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd957e4e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950107a80 con 0x7fd9500f6810 2021-11-20T07:38:35.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48086/0 (socket says 172.21.15.71:48086) 2021-11-20T07:38:35.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 -- 172.21.15.71:0/555115293 learned_addr learned my addr 172.21.15.71:0/555115293 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:35.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 -- 172.21.15.71:0/555115293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9500f6810 msgr2=0x7fd950107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9500f6810 0x7fd950107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:35.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.557+0000 7fd955bea700 1 -- 172.21.15.71:0/555115293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c005040 con 0x7fd9500f5570 2021-11-20T07:38:35.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd955bea700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd94c004e30 tx=0x7fd94c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd94c018070 con 0x7fd9500f5570 2021-11-20T07:38:35.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95010c4c0 con 0x7fd9500f5570 2021-11-20T07:38:35.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd94c013410 con 0x7fd9500f5570 2021-11-20T07:38:35.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd94c01c3c0 con 0x7fd9500f5570 2021-11-20T07:38:35.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.558+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95010ca80 con 0x7fd9500f5570 2021-11-20T07:38:35.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.559+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950027a00 con 0x7fd9500f5570 2021-11-20T07:38:35.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.559+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd94c01d420 con 0x7fd9500f5570 2021-11-20T07:38:35.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.559+0000 7fd9467fc700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 0x7fd93c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.559+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd94c077b60 con 0x7fd9500f5570 2021-11-20T07:38:35.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.559+0000 7fd9553e9700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 0x7fd93c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.560+0000 7fd9553e9700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 0x7fd93c05e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd940004a10 tx=0x7fd9400099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.563+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd94c04a820 con 0x7fd9500f5570 2021-11-20T07:38:35.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.716+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fd9500fee00 con 0x7fd9500f5570 2021-11-20T07:38:35.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.717+0000 7fd9467fc700 1 -- 172.21.15.71:0/555115293 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd94c04de40 con 0x7fd9500f5570 2021-11-20T07:38:35.719 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:35.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 msgr2=0x7fd93c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 0x7fd93c05e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd940004a10 tx=0x7fd9400099e0).stop 2021-11-20T07:38:35.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 msgr2=0x7fd950106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd94c004e30 tx=0x7fd94c009b30).stop 2021-11-20T07:38:35.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 shutdown_connections 2021-11-20T07:38:35.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd93c05c2a0 0x7fd93c05e760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9500f5570 0x7fd950106ea0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 --2- 172.21.15.71:0/555115293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9500f6810 0x7fd950107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:35.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 >> 172.21.15.71:0/555115293 conn(0x7fd9500f0cf0 msgr2=0x7fd9500fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 shutdown_connections 2021-11-20T07:38:35.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.719+0000 7fd957e4e700 1 -- 172.21.15.71:0/555115293 wait complete. 2021-11-20T07:38:35.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T07:38:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:35 smithi112 conmon[36062]: audit 2021-11-20T07:38:34.359440+0000 mon.smithi071 (mon.0) 530 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:35 smithi112 conmon[36062]: audit 2021-11-20T07:38:34.362340+0000 mon.smithi071 (mon.0) 531 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:38:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:35 smithi112 conmon[36062]: cluster 2021-11-20T07:38:34.365421+0000 mgr.smithi071.ynqmwn (mgr.14162) 198 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2021-11-20T07:38:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:35 smithi112 conmon[36062]: audit 2021-11-20T07:38:35.108173+0000 mon.smithi071 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.71:0/2069349217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T07:38:35.852 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 -- 172.21.15.71:0/1667924822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 msgr2=0x7f84800f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.853 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 --2- 172.21.15.71:0/1667924822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f84800f6900 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f8470004660 tx=0x7f8470009b30).stop 2021-11-20T07:38:35.853 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 -- 172.21.15.71:0/1667924822 shutdown_connections 2021-11-20T07:38:35.853 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 --2- 172.21.15.71:0/1667924822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84800f7780 0x7f84800f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.853 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 --2- 172.21.15.71:0/1667924822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f84800f6900 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:35.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 -- 172.21.15.71:0/1667924822 >> 172.21.15.71:0/1667924822 conn(0x7f84800f1c40 msgr2=0x7f84800f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:35.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 -- 172.21.15.71:0/1667924822 shutdown_connections 2021-11-20T07:38:35.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.847+0000 7f847f59e700 1 -- 172.21.15.71:0/1667924822 wait complete. 2021-11-20T07:38:35.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 Processor -- start 2021-11-20T07:38:35.855 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 -- start start 2021-11-20T07:38:35.855 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.855 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84800f7780 0x7f8480107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.855 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8480107930 con 0x7f84800f64e0 2021-11-20T07:38:35.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847f59e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8480108140 con 0x7f84800f7780 2021-11-20T07:38:35.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48096/0 (socket says 172.21.15.71:48096) 2021-11-20T07:38:35.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 -- 172.21.15.71:0/2652499195 learned_addr learned my addr 172.21.15.71:0/2652499195 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:35.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 -- 172.21.15.71:0/2652499195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84800f7780 msgr2=0x7f8480107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:35.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84800f7780 0x7f8480107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:35.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 -- 172.21.15.71:0/2652499195 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470005040 con 0x7f84800f64e0 2021-11-20T07:38:35.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.848+0000 7f847e59c700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8470004e30 tx=0x7f8470009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8470018070 con 0x7f84800f64e0 2021-11-20T07:38:35.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8470013410 con 0x7f84800f64e0 2021-11-20T07:38:35.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8480108360 con 0x7f84800f64e0 2021-11-20T07:38:35.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f847001c3c0 con 0x7f84800f64e0 2021-11-20T07:38:35.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84801088c0 con 0x7f84800f64e0 2021-11-20T07:38:35.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.849+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84800f58d0 con 0x7f84800f64e0 2021-11-20T07:38:35.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.850+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f847001d420 con 0x7f84800f64e0 2021-11-20T07:38:35.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.850+0000 7f846effd700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 0x7f846805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:35.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.850+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8470077ac0 con 0x7f84800f64e0 2021-11-20T07:38:35.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.850+0000 7f847dd9b700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 0x7f846805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:35.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.850+0000 7f847dd9b700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 0x7f846805e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8474004a10 tx=0x7f84740099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:35.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:35.854+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f847004a780 con 0x7f84800f64e0 2021-11-20T07:38:36.005 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.004+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8480045ad0 con 0x7f84800f64e0 2021-11-20T07:38:36.006 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.004+0000 7f846effd700 1 -- 172.21.15.71:0/2652499195 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f847004dda0 con 0x7f84800f64e0 2021-11-20T07:38:36.006 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2021-11-20T07:38:36.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.005+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 msgr2=0x7f846805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.005+0000 7f847f59e700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 0x7f846805e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8474004a10 tx=0x7f84740099e0).stop 2021-11-20T07:38:36.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.005+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 msgr2=0x7f8480106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.005+0000 7f847f59e700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8470004e30 tx=0x7f8470009b30).stop 2021-11-20T07:38:36.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 shutdown_connections 2021-11-20T07:38:36.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f846805c2b0 0x7f846805e770 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.008 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84800f64e0 0x7f8480106e40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 --2- 172.21.15.71:0/2652499195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84800f7780 0x7f8480107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:36.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 >> 172.21.15.71:0/2652499195 conn(0x7f84800f1c40 msgr2=0x7f84800fb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:36.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 shutdown_connections 2021-11-20T07:38:36.009 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.006+0000 7f847f59e700 1 -- 172.21.15.71:0/2652499195 wait complete. 2021-11-20T07:38:36.019 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T07:38:36.019 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:38:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 -- 172.21.15.71:0/1988383017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b740f96c0 msgr2=0x7f4b740fbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 --2- 172.21.15.71:0/1988383017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b740f96c0 0x7f4b740fbb10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f4b68004660 tx=0x7f4b68009b30).stop 2021-11-20T07:38:36.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 -- 172.21.15.71:0/1988383017 shutdown_connections 2021-11-20T07:38:36.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 --2- 172.21.15.71:0/1988383017 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b740f96c0 0x7f4b740fbb10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 --2- 172.21.15.71:0/1988383017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b74060140 0x7f4b740f9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 -- 172.21.15.71:0/1988383017 >> 172.21.15.71:0/1988383017 conn(0x7f4b740f0ba0 msgr2=0x7f4b740f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:36.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 -- 172.21.15.71:0/1988383017 shutdown_connections 2021-11-20T07:38:36.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.139+0000 7f4b7359e700 1 -- 172.21.15.71:0/1988383017 wait complete. 2021-11-20T07:38:36.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 Processor -- start 2021-11-20T07:38:36.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 -- start start 2021-11-20T07:38:36.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b740f96c0 0x7f4b740f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7410a9f0 con 0x7f4b74060140 2021-11-20T07:38:36.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7359e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7410ab30 con 0x7f4b740f96c0 2021-11-20T07:38:36.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48108/0 (socket says 172.21.15.71:48108) 2021-11-20T07:38:36.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 -- 172.21.15.71:0/3498127869 learned_addr learned my addr 172.21.15.71:0/3498127869 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:36.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b71d9b700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b740f96c0 0x7f4b740f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 -- 172.21.15.71:0/3498127869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b740f96c0 msgr2=0x7f4b740f5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b740f96c0 0x7f4b740f5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.140+0000 7f4b7259c700 1 -- 172.21.15.71:0/3498127869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68005040 con 0x7f4b74060140 2021-11-20T07:38:36.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b7259c700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f4b6c007fd0 tx=0x7f4b6c004ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6c00b4f0 con 0x7f4b74060140 2021-11-20T07:38:36.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b740f55f0 con 0x7f4b74060140 2021-11-20T07:38:36.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b6c00f040 con 0x7f4b74060140 2021-11-20T07:38:36.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6c013a00 con 0x7f4b74060140 2021-11-20T07:38:36.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.141+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b740f5bb0 con 0x7f4b74060140 2021-11-20T07:38:36.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.142+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74045ad0 con 0x7f4b74060140 2021-11-20T07:38:36.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.142+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4b6c01a030 con 0x7f4b74060140 2021-11-20T07:38:36.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.142+0000 7f4b62ffd700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 0x7f4b5c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.142+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4b6c075c00 con 0x7f4b74060140 2021-11-20T07:38:36.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.142+0000 7f4b71d9b700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 0x7f4b5c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.143+0000 7f4b71d9b700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 0x7f4b5c05e7c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4b68009f50 tx=0x7f4b68009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.146+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b6c07b020 con 0x7f4b74060140 2021-11-20T07:38:36.332 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.330+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b740f5e30 con 0x7f4b74060140 2021-11-20T07:38:36.333 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.330+0000 7f4b62ffd700 1 -- 172.21.15.71:0/3498127869 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f4b6c018030 con 0x7f4b74060140 2021-11-20T07:38:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:38:36.335 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/1338231265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1338231265},{"type":"v1","addr":"172.21.15.71:6829","nonce":1338231265}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24267,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3800926744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3800926744},{"type":"v1","addr":"172.21.15.112:6827","nonce":3800926744}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:38:20.440630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14430,"mds_1":24253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14430":{"gid":14430,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.71:6827/657608204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":657608204},{"type":"v1","addr":"172.21.15.71:6827","nonce":657608204}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24253":{"gid":24253,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":7,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6825/3641386819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3641386819},{"type":"v1","addr":"172.21.15.112:6825","nonce":3641386819}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:38:36.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 msgr2=0x7f4b5c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.337 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 0x7f4b5c05e7c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4b68009f50 tx=0x7f4b68009b30).stop 2021-11-20T07:38:36.338 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 msgr2=0x7f4b740f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.339 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f4b6c007fd0 tx=0x7f4b6c004ee0).stop 2021-11-20T07:38:36.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 shutdown_connections 2021-11-20T07:38:36.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4b5c05c300 0x7f4b5c05e7c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.340 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b74060140 0x7f4b740f6bc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 --2- 172.21.15.71:0/3498127869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b740f96c0 0x7f4b740f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 >> 172.21.15.71:0/3498127869 conn(0x7f4b740f0ba0 msgr2=0x7f4b740f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:36.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 shutdown_connections 2021-11-20T07:38:36.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.332+0000 7f4b7359e700 1 -- 172.21.15.71:0/3498127869 wait complete. 2021-11-20T07:38:36.342 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 12 2021-11-20T07:38:36.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:38:20.440630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14430, 'mds_1': 24253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14430': {'gid': 14430, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.71:6827/657608204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 657608204}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 657608204}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24253': {'gid': 24253, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6825/3641386819', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 3641386819}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 3641386819}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:38:36.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-20T07:38:36.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi071.xtyomb status 2021-11-20T07:38:36.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- 172.21.15.71:0/1288439906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f7b40 msgr2=0x7f29b80f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 --2- 172.21.15.71:0/1288439906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f7b40 0x7f29b80f7fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f29a8004660 tx=0x7f29a8009b30).stop 2021-11-20T07:38:36.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- 172.21.15.71:0/1288439906 shutdown_connections 2021-11-20T07:38:36.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 --2- 172.21.15.71:0/1288439906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f7b40 0x7f29b80f7fa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 --2- 172.21.15.71:0/1288439906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29b80f68a0 0x7f29b80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- 172.21.15.71:0/1288439906 >> 172.21.15.71:0/1288439906 conn(0x7f29b80f1fc0 msgr2=0x7f29b80f4420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:36.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- 172.21.15.71:0/1288439906 shutdown_connections 2021-11-20T07:38:36.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- 172.21.15.71:0/1288439906 wait complete. 2021-11-20T07:38:36.472 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 Processor -- start 2021-11-20T07:38:36.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- start start 2021-11-20T07:38:36.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f68a0 0x7f29b8107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.473 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29b80f7b40 0x7f29b8107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8107b20 con 0x7f29b80f68a0 2021-11-20T07:38:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.465+0000 7f29bfc05700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8107c60 con 0x7f29b80f7b40 2021-11-20T07:38:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f68a0 0x7f29b8107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f68a0 0x7f29b8107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48116/0 (socket says 172.21.15.71:48116) 2021-11-20T07:38:36.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 -- 172.21.15.71:0/3797688236 learned_addr learned my addr 172.21.15.71:0/3797688236 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:36.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 -- 172.21.15.71:0/3797688236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29b80f7b40 msgr2=0x7f29b8107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/3797688236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29b80f7b40 0x7f29b8107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:38:36.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 -- 172.21.15.71:0/3797688236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8005040 con 0x7f29b80f68a0 2021-11-20T07:38:36.475 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/3797688236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29b80f68a0 0x7f29b8107080 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f29b400c760 tx=0x7f29b4004cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29ae7fc700 1 -- 172.21.15.71:0/3797688236 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b40106f0 con 0x7f29b80f68a0 2021-11-20T07:38:36.477 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29ae7fc700 1 -- 172.21.15.71:0/3797688236 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29b4010d00 con 0x7f29b80f68a0 2021-11-20T07:38:36.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bfc05700 1 -- 172.21.15.71:0/3797688236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b810c700 con 0x7f29b80f68a0 2021-11-20T07:38:36.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29ae7fc700 1 -- 172.21.15.71:0/3797688236 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b4012d10 con 0x7f29b80f68a0 2021-11-20T07:38:36.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.466+0000 7f29bfc05700 1 -- 172.21.15.71:0/3797688236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b810ccc0 con 0x7f29b80f68a0 2021-11-20T07:38:36.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.467+0000 7f29ae7fc700 1 -- 172.21.15.71:0/3797688236 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f29b4010070 con 0x7f29b80f68a0 2021-11-20T07:38:36.480 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.468+0000 7f29ae7fc700 1 --2- 172.21.15.71:0/3797688236 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29a405c2a0 0x7f29a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.480 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.468+0000 7f29ae7fc700 1 -- 172.21.15.71:0/3797688236 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f29b4075e80 con 0x7f29b80f68a0 2021-11-20T07:38:36.481 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.468+0000 7f29bd1a0700 1 --2- 172.21.15.71:0/3797688236 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29a405c2a0 0x7f29a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.481 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.468+0000 7f29bd1a0700 1 --2- 172.21.15.71:0/3797688236 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29a405c2a0 0x7f29a405e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f29a8009f50 tx=0x7f29a8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 Processor -- start 2021-11-20T07:38:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 -- start start 2021-11-20T07:38:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d607835fe0 0x55d607836d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6077744d0 con 0x55d6076a31d0 2021-11-20T07:38:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29d17b0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6077883e0 con 0x55d607835fe0 2021-11-20T07:38:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48122/0 (socket says 172.21.15.71:48122) 2021-11-20T07:38:36.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 -- 172.21.15.71:0/1315667813 learned_addr learned my addr 172.21.15.71:0/1315667813 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:36.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd1a0700 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d607835fe0 0x55d607836d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.491 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 -- 172.21.15.71:0/1315667813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d607835fe0 msgr2=0x55d607836d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d607835fe0 0x55d607836d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 -- 172.21.15.71:0/1315667813 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6078043c0 con 0x55d6076a31d0 2021-11-20T07:38:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.470+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f29b4011040 tx=0x7f29b4048b40).ready entity=mon.0 client_cookie=462d36306bcdca1a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f299b7fe700 1 -- 172.21.15.71:0/1315667813 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b4048f10 con 0x55d6076a31d0 2021-11-20T07:38:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f299b7fe700 1 -- 172.21.15.71:0/1315667813 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29b4049070 con 0x55d6076a31d0 2021-11-20T07:38:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f299b7fe700 1 -- 172.21.15.71:0/1315667813 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b40488c0 con 0x55d6076a31d0 2021-11-20T07:38:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 -- 172.21.15.71:0/1315667813 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 msgr2=0x55d607835af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:38:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f29b4011040 tx=0x7f29b4048b40).stop 2021-11-20T07:38:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 -- 172.21.15.71:0/1315667813 shutdown_connections 2021-11-20T07:38:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d607835fe0 0x55d607836d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 --2- 172.21.15.71:0/1315667813 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d6076a31d0 0x55d607835af0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 -- 172.21.15.71:0/1315667813 >> 172.21.15.71:0/1315667813 conn(0x55d6077fb9b0 msgr2=0x55d607830fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:38:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 -- 172.21.15.71:0/1315667813 shutdown_connections 2021-11-20T07:38:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.471+0000 7f29d17b0b80 1 -- 172.21.15.71:0/1315667813 wait complete. 2021-11-20T07:38:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 -- start start 2021-11-20T07:38:36.498 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 Processor -- start 2021-11-20T07:38:36.498 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6076a31d0 0x55d60783e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d607835fe0 0x55d60783ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.499 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d607788520 con 0x55d607835fe0 2021-11-20T07:38:36.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d607788aa0 con 0x55d6076a31d0 2021-11-20T07:38:36.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 10 client.0 ll_register_callbacks cb 0x55d6077f5700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T07:38:36.500 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d607835fe0 0x55d60783ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.501 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d607835fe0 0x55d60783ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48128/0 (socket says 172.21.15.71:48128) 2021-11-20T07:38:36.501 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 -- 172.21.15.71:0/498837014 learned_addr learned my addr 172.21.15.71:0/498837014 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:38:36.502 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6076a31d0 0x55d60783e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:36.502 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6076a31d0 msgr2=0x55d60783e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T07:38:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d6076a31d0 0x55d60783e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:38:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 -- 172.21.15.71:0/498837014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b810ccc0 con 0x55d607835fe0 2021-11-20T07:38:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29bd1a0700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d607835fe0 0x55d60783ed90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f29a80093d0 tx=0x7f29a801a000).ready entity=mon.0 client_cookie=ea573db861d10358 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:38:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f2998ff9700 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2021-11-20T07:38:36.506 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f2998ff9700 1 -- 172.21.15.71:0/498837014 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a8024070 con 0x55d607835fe0 2021-11-20T07:38:36.506 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f2998ff9700 1 -- 172.21.15.71:0/498837014 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f29a801f8e0 con 0x55d607835fe0 2021-11-20T07:38:36.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.472+0000 7f29d17b0b80 1 -- 172.21.15.71:0/498837014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d607803ce0 con 0x55d607835fe0 2021-11-20T07:38:36.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f2998ff9700 1 -- 172.21.15.71:0/498837014 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29a800e030 con 0x55d607835fe0 2021-11-20T07:38:36.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 10 client.14546 fetch_fsmap learned FSMap version 12 2021-11-20T07:38:36.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 1 -- 172.21.15.71:0/498837014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x55d607777630 con 0x55d607835fe0 2021-11-20T07:38:36.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f2998ff9700 1 -- 172.21.15.71:0/498837014 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55d607803ce0 con 0x55d607835fe0 2021-11-20T07:38:36.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f2998ff9700 1 -- 172.21.15.71:0/498837014 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 12) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f29a8012b00 con 0x55d607835fe0 2021-11-20T07:38:36.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 10 client.14546 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T07:38:36.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 10 client.14546 resolve_mds: resolved name 'cephfs.smithi071.xtyomb' to daemon mds.cephfs.smithi071.xtyomb 2021-11-20T07:38:36.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 20 client.14546 populate_metadata read hostname 'smithi071' 2021-11-20T07:38:36.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T07:38:36.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 4 client.14546 mds_command: new command op to 14430 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T07:38:36.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29d17b0b80 1 -- 172.21.15.71:0/498837014 --> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d60777edf0 con 0x55d607847670 2021-11-20T07:38:36.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:36.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:36.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.473+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T07:38:36.675 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.674+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:36.676 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.674+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T07:38:36.676 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:36.674+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:36.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: audit 2021-11-20T07:38:35.419476+0000 mon.smithi112 (mon.1) 15 : 2021-11-20T07:38:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/2933329057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: audit 2021-11-20T07:38:35.718559+0000 mon.smithi071 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.71:0/555115293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: audit 2021-11-20T07:38:36.005540+0000 mon.smithi071 (mon.0) 534 : audit [DBG] 2021-11-20T07:38:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/2652499195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: audit 2021-11-20T07:38:36.331524 2021-11-20T07:38:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.71:0/3498127869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:36 smithi112 conmon[36062]: 2021-11-20T07:38:36.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:36 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:38:36.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:36 smithi071 conmon[33687]: 38:35.419476+0000 mon.smithi112 (mon.1) 15 : audit [DBG] from='client.? 172.21.15.71:0/2933329057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:36 smithi071 conmon[33687]: audit 2021-11-20T07:38:35.718559+0000 mon.smithi071 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.71:0/555115293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:36 smithi071 conmon[33687]: audit 2021-11-20T07:38:36.005540+0000 mon.smithi071 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.71:0/2652499195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T07:38:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:36 smithi071 conmon[33687]: audit 2021-11-20T07:38:36.331524+0000 mon.smithi071 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.71:0/3498127869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:38:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.075+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.075+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T07:38:37.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.075+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:37.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:37 smithi071 conmon[33687]: cluster 2021-11-20T07:38:36.365806+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:38:37.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:37 smithi071 conmon[33687]: ) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:37 smithi112 conmon[36062]: cluster 2021-11-20T07:38:36.365806+0000 mgr.smithi071.ynqmwn (mgr.14162) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:37.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.876+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:37.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.876+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T07:38:37.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:37.876+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:39.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:39.477+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:39.478 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:39.477+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T07:38:39.479 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:39.477+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:39 smithi071 conmon[33687]: cluster 2021-11-20T07:38:38.366320+0000 mgr.smithi071.ynqmwn (mgr.14162) 200 : 2021-11-20T07:38:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:39 smithi071 conmon[33687]: cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:39 smithi112 conmon[36062]: cluster 2021-11-20T07:38:38.366320+0000 mgr.smithi071.ynqmwn (mgr.14162) 200 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:41 smithi112 conmon[36062]: cluster 2021-11-20T07:38:40.366783+0000 mgr.smithi071.ynqmwn (mgr.14162) 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:41.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:41 smithi071 conmon[33687]: cluster 2021-11-20T07:38:40.366783+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:38:41.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:41 smithi071 conmon[33687]: 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:42.681 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:42.681+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:38:42.682 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:42.681+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 6.400000 2021-11-20T07:38:42.682 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:42.681+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:38:43.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:43 smithi071 conmon[33687]: cluster 2021-11-20T07:38:42.367299+0000 mgr.smithi071.ynqmwn (mgr.14162) 202 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:43.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:43 smithi071 conmon[33687]: 2021-11-20T07:38:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:43 smithi112 conmon[36062]: cluster 2021-11-20T07:38:42.367299+0000 mgr.smithi071.ynqmwn (mgr.14162) 202 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:45.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:45 smithi112 conmon[36062]: cluster 2021-11-20T07:38:44.367773+0000 mgr.smithi071.ynqmwn (mgr.14162) 203 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:45.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:45 smithi071 conmon[33687]: cluster 2021-11-20T07:38:44.367773+0000 mgr.smithi071.ynqmwn (mgr.14162) 203 : cluster 2021-11-20T07:38:45.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:45 smithi071 conmon[33687]: [DBG] pgmap v164: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2021-11-20T07:38:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:47 smithi112 conmon[36062]: cluster 2021-11-20T07:38:46.368177+0000 mgr.smithi071.ynqmwn (mgr.14162) 204 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T07:38:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:47 smithi112 conmon[36062]: 2021-11-20T07:38:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:47 smithi071 conmon[33687]: cluster 2021-11-20T07:38:46.368177+0000 mgr.smithi071.ynqmwn (mgr.14162) 204 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T07:38:49.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:49.087+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:38:49.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:49.087+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:38:49.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:49.087+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:38:49.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:49.087+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:38:49.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:38:49.087+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 12.800000 2021-11-20T07:38:49.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:49 smithi112 conmon[36062]: cluster 2021-11-20T07:38:48.368701+0000 mgr.smithi071.ynqmwn (mgr.14162) 205 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:49.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:49 smithi112 conmon[36062]: 2021-11-20T07:38:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:49 smithi071 conmon[33687]: cluster 2021-11-20T07:38:48.368701+0000 mgr.smithi071.ynqmwn (mgr.14162) 205 : cluster 2021-11-20T07:38:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:49 smithi071 conmon[33687]: [DBG] pgmap v166: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:50.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021-11-20T07:38:50.312841+0000 mon.smithi071 (mon.0) 536 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:38:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021-11-20T07:38:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: 2021-11-20T07:38:50.317523+0000 mon.smithi071 (mon.0) 537 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T07:38:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021-11-20T07:38:50.368016+0000 mon.smithi071 (mon.0) 538 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:38:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:38:50.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: -11-20T07:38:50.368305+0000 mon.smithi071 (mon.0) 539 : cluster [WRN] 2021-11-20T07:38:50.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T07:38:50.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021-11-20T07:38:50.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: 2021-11-20T07:38: 2021-11-20T07:38:50.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: 50.372383+0000 mon.smithi071 (mon.0) 2021-11-20T07:38:50.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: 540 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:boot 2021-11-20T07:38:50.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:38:50.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: -20T07:38:50.372440+0000 mon.smithi071 2021-11-20T07:38:50.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: (mon.0) 541 : cluster [DBG] 2021-11-20T07:38:50.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:50.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:38:50.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: 11-20T07:38:50.372998 2021-11-20T07:38:50.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 542 2021-11-20T07:38:50.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:50 smithi112 conmon[36062]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:38:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.312841+0000 mon.smithi071 (mon.0) 536 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:38:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.317523+0000 mon.smithi071 (mon.0) 537 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T07:38:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.368016+0000 mon.smithi071 (mon.0) 538 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:38:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.368305+0000 mon.smithi071 (mon.0) 539 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T07:38:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.372383+0000 mon.smithi071 (mon 2021-11-20T07:38:50.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: .0) 540 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:boot 2021-11-20T07:38:50.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.372440+0000 mon.smithi071 (mon. 2021-11-20T07:38:50.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: 0) 541 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:50.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: audit 2021-11-20T07:38:50.372998+0000 mon.smithi071 (mon.0 2021-11-20T07:38:50.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:50 smithi071 conmon[33687]: ) 542 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:38:51.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:51 smithi112 conmon[36062]: cluster 2021-11-20T07:38:50.369049+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:38:51.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:51 smithi112 conmon[36062]: 206 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:51.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:51 smithi112 conmon[36062]: cluster 2021-11-20T07:38:51.377574+0000 mon.smithi071 (mon.0) 543 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:resolve 2021-11-20T07:38:51.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:51 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:38:51.474 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:51 smithi112 conmon[36062]: -11-20T07:38:51.377648+0000 mon.smithi071 (mon.0) 544 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:51.514 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:51 smithi071 conmon[33687]: cluster 2021-11-20T07:38:50.369049+0000 mgr.smithi071.ynqmwn (mgr.14162) 206 : cluster [DBG] 2021-11-20T07:38:51.515 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:51 smithi071 conmon[33687]: pgmap v168: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:38:51.515 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:51 smithi071 conmon[33687]: cluster 2021-11-20T07:38:51.377574+0000 mon.smithi071 (mon.0) 543 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:resolve 2021-11-20T07:38:51.515 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:51 smithi071 conmon[33687]: cluster 2021-11-20T07:38:51.377648+0000 mon.smithi071 (mon.0) 544 : 2021-11-20T07:38:51.516 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:51 smithi071 conmon[33687]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:52 smithi112 conmon[36062]: cluster 2021-11-20T07:38:52.369521+0000 mgr.smithi071.ynqmwn (mgr.14162) 207 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:38:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:52 smithi071 conmon[33687]: cluster 2021-11-20T07:38:52.369521+0000 mgr.smithi071.ynqmwn (mgr.14162) 207 : cluster [DBG] 2021-11-20T07:38:52.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:52 smithi071 conmon[33687]: pgmap v169: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:38:55.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:55 smithi071 conmon[33687]: cluster 2021-11-20T07:38:54.367477+0000 mon.smithi071 (mon.0) 545 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:standby 2021-11-20T07:38:55.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:55 smithi071 conmon[33687]: cluster 2021-11-20T07:38:54.367545+0000 mon.smithi071 (mon.0) 546 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:55.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:55 smithi071 conmon[33687]: cluster 2021-11-20T07:38:54.369965+0000 mgr.smithi071.ynqmwn (mgr.14162) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:38:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:55 smithi112 conmon[36062]: cluster 2021-11-20T07:38:54.367477+0000 mon.smithi071 (mon.0) 545 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:standby 2021-11-20T07:38:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:55 smithi112 conmon[36062]: cluster 2021-11-20T07:38:54.367545+0000 2021-11-20T07:38:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:55 smithi112 conmon[36062]: mon.smithi071 (mon.0) 546 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:38:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:55 smithi112 conmon[36062]: cluster 2021-11-20T07:38:54.369965+0000 mgr.smithi071.ynqmwn (mgr.14162) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:38:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:57 smithi112 conmon[36062]: cluster 2021-11-20T07:38:56.370440+0000 mgr.smithi071.ynqmwn (mgr.14162) 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:38:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:57 smithi071 conmon[33687]: cluster 2021-11-20T07:38:56.370440+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:38:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:57 smithi071 conmon[33687]: .14162) 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:38:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:38:59 smithi071 conmon[33687]: cluster 2021-11-20T07:38:58.370878+0000 mgr.smithi071.ynqmwn (mgr.14162) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:38:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:38:59 smithi112 conmon[36062]: cluster 2021-11-20T07:38:58.370878+0000 mgr.smithi071.ynqmwn (mgr.14162) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:01 smithi112 conmon[36062]: cluster 2021-11-20T07:39:00.371319+0000 mgr.smithi071.ynqmwn (mgr.14162) 211 : cluster [DBG] 2021-11-20T07:39:01.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:01 smithi112 conmon[36062]: pgmap v173: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:01 smithi071 conmon[33687]: cluster 2021-11-20T07:39:00.371319+0000 mgr.smithi071.ynqmwn (mgr.14162) 211 : cluster 2021-11-20T07:39:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:01 smithi071 conmon[33687]: [DBG] pgmap v173: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:01.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:01.888+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:39:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:01.889+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:39:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:01.889+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:39:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:01.889+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:39:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:01.889+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:39:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:03 smithi112 conmon[36062]: cluster 2021-11-20T07:39:02.371848+0000 mgr.smithi071.ynqmwn (mgr.14162) 212 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:39:03.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:03 smithi071 conmon[33687]: cluster 2021-11-20T07:39:02.371848+0000 mgr.smithi071.ynqmwn (mgr.14162) 212 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:39:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: cluster 2021-11-20T07:39:04.372287+0000 mgr.smithi071.ynqmwn (mgr.14162) 213 : cluster [DBG] 2021-11-20T07:39:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: pgmap v175: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:39:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: cluster 2021-11-20T07:39:05.371327+0000 mon.smithi071 (mon.0) 547 : cluster [WRN] Replacing daemon mds.cephfs.smithi112.bxcrgi as rank 1 with standby daemon mds.cephfs.smithi071.xtyomb 2021-11-20T07:39:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: cluster 2021-11-20T07:39:05.371418+0000 mon.smithi071 (mon. 2021-11-20T07:39:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: 0) 548 : cluster [INF] MDS daemon mds.cephfs.smithi112.bxcrgi is removed because it is dead or otherwise unavailable. 2021-11-20T07:39:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: cluster 2021-11-20T07:39:05. 2021-11-20T07:39:05.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: 376542+0000 mon.smithi071 (mon.0) 549 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T07:39:05.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:39:05.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: 20T07:39:05.380010+0000 mon.smithi071 (mon 2021-11-20T07:39:05.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:05 smithi112 conmon[36062]: .0) 550 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 1 up:standby 2021-11-20T07:39:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster 2021-11-20T07:39:04.372287+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:39:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: 213 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:39:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster 2021-11-20T07:39:05.371327+0000 mon.smithi071 (mon.0) 547 : cluster [WRN] Replacing daemon mds.cephfs.smithi112.bxcrgi as rank 1 with standby daemon mds.cephfs.smithi071.xtyomb 2021-11-20T07:39:05.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster 2021-11-20T07:39:05.371418+0000 mon.smithi071 (mon. 2021-11-20T07:39:05.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: 0) 548 : cluster [INF] MDS daemon mds.cephfs.smithi112.bxcrgi is removed because it is dead or otherwise unavailable. 2021-11-20T07:39:05.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:39:05.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: 39:05.376542+0000 mon.smithi071 (mon.0) 549 2021-11-20T07:39:05.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T07:39:05.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:39:05.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: 39:05.380010+0000 mon.smithi071 (mon.0) 550 : 2021-11-20T07:39:05.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:05 smithi071 conmon[33687]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 1 up:standby 2021-11-20T07:39:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: cluster 2021-11-20T07:39:06.372650+0000 mgr.smithi071.ynqmwn (mgr.14162) 214 2021-11-20T07:39:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:39:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: cluster 2021-11-20T07:39:06.381477+0000 mon.smithi071 (mon.0) 551 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:resolve 2021-11-20T07:39:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: cluster 2021-11-20T07:39:06.381531+0000 mon.smithi071 2021-11-20T07:39:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: (mon.0) 552 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:boot 2021-11-20T07:39:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:39:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: -11-20T07:39:06.381586+0000 mon.smithi071 2021-11-20T07:39:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: (mon.0) 553 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:39:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: audit 2021 2021-11-20T07:39:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: -11-20T07:39:06.381715+0000 2021-11-20T07:39:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:07 smithi112 conmon[36062]: mon.smithi071 (mon.0) 554 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:39:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: cluster 2021-11-20T07:39:06.372650+0000 mgr.smithi071.ynqmwn (mgr.14162) 214 : cluster [DBG] 2021-11-20T07:39:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: pgmap v177: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:39:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: cluster 2021-11-20T07:39:06.381477+0000 mon.smithi071 (mon.0) 551 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:resolve 2021-11-20T07:39:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: cluster 2021-11-20T07:39:06.381531+0000 mon.smithi071 (mon.0) 552 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:boot 2021-11-20T07:39:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: cluster 2021-11-20T07:39:06.381586+0000 mon.smithi071 (mon.0) 553 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:39:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: audit 2021 2021-11-20T07:39:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:07 smithi071 conmon[33687]: -11-20T07:39:06.381715+0000 mon.smithi071 (mon.0) 554 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:39:08.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: cluster 2021-11-20T07:39:07.383477+0000 2021-11-20T07:39:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: mon.smithi071 (mon.0) 555 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:reconnect 2021-11-20T07:39:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: cluster 2021-11-20T07:39:07.383522+0000 mon.smithi071 (mon.0) 556 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:reconnect 2021-11-20T07:39:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: cluster 2021-11-20T07:39:07.383604+0000 mon.smithi071 (mon.0) 557 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:39:08.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: cluster 2021-11-20T07:39:07.408098+0000 2021-11-20T07:39:08.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: mds.cephfs.smithi071.zaejme (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:39:08.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: cluster 2021-11-20T07:39:07.408106+0000 2021-11-20T07:39:08.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:08 smithi071 conmon[33687]: mds.cephfs.smithi071.xtyomb (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:39:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: cluster 2021-11-20T07:39:07.383477+0000 mon.smithi071 (mon.0) 2021-11-20T07:39:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: 555 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:reconnect 2021-11-20T07:39:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: cluster 2021-11-20T07:39:07.383522+0000 mon.smithi071 (mon.0) 556 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:reconnect 2021-11-20T07:39:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: cluster 2021-11-20T07:39:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: 2021-11-20T07:39:07.383604+0000 mon.smithi071 (mon.0) 557 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:39:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:39:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: 07:39:07.408098+0000 mds.cephfs.smithi071.zaejme (mds.0 2021-11-20T07:39:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: ) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:39:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:39:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: -20T07:39:07.408106+0000 mds.cephfs.smithi071.xtyomb 2021-11-20T07:39:08.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:08 smithi112 conmon[36062]: (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:39:09.472 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: cluster 2021-11-20T07:39:08.373062+0000 mgr.smithi071.ynqmwn (mgr.14162) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:39:09.473 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: cluster 2021-11-20T07:39:09.473 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: 2021-11-20T07:39:08.395145+0000 mon.smithi071 (mon.0) 558 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:rejoin 2021-11-20T07:39:09.473 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: cluster 2021-11-20T07:39:08.395178+0000 mon.smithi071 (mon.0) 559 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:rejoin 2021-11-20T07:39:09.474 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: cluster 2021-11-20T07:39:09.474 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:09 smithi071 conmon[33687]: 2021-11-20T07:39:08.395239+0000 mon.smithi071 (mon.0) 560 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: cluster 2021-11-20T07:39:08.373062+0000 mgr.smithi071.ynqmwn (mgr.14162) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:39:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: cluster 2021-11-20T07:39:08.395145+0000 mon.smithi071 2021-11-20T07:39:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: (mon.0) 558 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1338231265,v1:172.21.15.71:6829/1338231265] up:rejoin 2021-11-20T07:39:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: cluster 2021-11-20T07:39:08.395178+0000 mon.smithi071 (mon.0) 559 : cluster [DBG] mds.? [v2:172.21.15.71:6826/3642873752,v1:172.21.15.71:6827/3642873752] up:rejoin 2021-11-20T07:39:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: cluster 2021-11-20T07:39:08.395239+0000 mon.smithi071 (mon.0) 2021-11-20T07:39:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:09 smithi112 conmon[36062]: 560 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:11.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:11 smithi071 conmon[33687]: cluster 2021-11-20T07:39:10.373491+0000 mgr.smithi071.ynqmwn (mgr.14162) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:11.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:11 smithi071 conmon[33687]: 2021-11-20T07:39:11.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:11 smithi071 conmon[33687]: cluster 2021-11-20T07:39:10.400528+0000 mon.smithi071 (mon.0) 561 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:standby 2021-11-20T07:39:11.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:11 smithi071 conmon[33687]: cluster 2021-11-20T07:39:10.400591 2021-11-20T07:39:11.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:11 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 562 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:11 smithi112 conmon[36062]: cluster 2021-11-20T07:39:10.373491+0000 mgr.smithi071.ynqmwn (mgr.14162) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:11 smithi112 conmon[36062]: cluster 2021-11-20T07:39:10.400528+0000 mon.smithi071 (mon.0) 561 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:standby 2021-11-20T07:39:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:11 smithi112 conmon[36062]: cluster 2021-11-20T07:39:10.400591+0000 mon.smithi071 (mon.0) 562 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:12 smithi112 conmon[36062]: cluster 2021-11-20T07:39:12.374228+0000 mgr.smithi071.ynqmwn (mgr.14162) 217 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:12 smithi071 conmon[33687]: cluster 2021-11-20T07:39:12.374228+0000 mgr.smithi071.ynqmwn (mgr.14162) 217 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:14 smithi112 conmon[36062]: cluster 2021-11-20T07:39:14.374623+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:39:14.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:14 smithi112 conmon[36062]: 218 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:14 smithi071 conmon[33687]: cluster 2021-11-20T07:39:14.374623+0000 mgr.smithi071.ynqmwn (mgr.14162) 218 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T07:39:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:16 smithi112 conmon[36062]: cluster 2021-11-20T07:39:16.375260+0000 mgr.smithi071.ynqmwn (mgr.14162) 219 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 930 B/s rd, 0 op/s 2021-11-20T07:39:16.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:16 smithi071 conmon[33687]: cluster 2021-11-20T07:39:16.375260+0000 mgr.smithi071.ynqmwn (mgr.14162) 219 : cluster [DBG] 2021-11-20T07:39:16.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:16 smithi071 conmon[33687]: pgmap v182: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 930 B/s rd, 0 op/s 2021-11-20T07:39:16.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:16.897+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:39:16.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:16.897+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:39:16.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:16.897+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:39:16.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:16.897+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:39:16.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:16.897+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:39:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:18 smithi112 conmon[36062]: cluster 2021-11-20T07:39:18.375734+0000 mgr.smithi071.ynqmwn (mgr.14162) 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T07:39:18.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:18 smithi071 conmon[33687]: cluster 2021-11-20T07:39:18.375734+0000 mgr.smithi071.ynqmwn (mgr.14162) 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T07:39:19.548 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:19 smithi071 conmon[33687]: audit 2021-11-20T07:39:18.934747+0000 mon.smithi071 (mon.0) 563 : audit 2021-11-20T07:39:19.548 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:19 smithi071 conmon[33687]: [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:39:19.548 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:19 smithi071 conmon[33687]: audit 2021-11-20T07:39:18.935288+0000 mon.smithi071 (mon.0) 564 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:39:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:19 smithi112 conmon[36062]: audit 2021-11-20T07:39:18.934747+0000 mon.smithi071 (mon.0) 563 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:39:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:19 smithi112 conmon[36062]: audit 2021-11-20T07:39:18.935288+0000 mon.smithi071 (mon.0) 564 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:39:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:19.481774+0000 mon.smithi071 (mon.0) 565 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:39:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:19.489606+0000 2021-11-20T07:39:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: mon.smithi071 (mon.0) 566 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T07:39:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:20.374583+0000 mon.smithi071 (mon.0) 567 : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:39:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:20.376064+0000 mgr.smithi071.ynqmwn (mgr.14162) 221 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:39:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:20.379540+0000 mon.smithi071 (mon.0) 568 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:boot 2021-11-20T07:39:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: cluster 2021-11-20T07:39:20.379610+0000 mon.smithi071 (mon.0) 569 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:20 smithi112 conmon[36062]: audit 2021-11-20T07:39:20.379925+0000 mon.smithi071 (mon.0) 570 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:39:20.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11-20T07:39:19.481774+0000 mon.smithi071 (mon.0) 565 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:39:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11-20T07:39:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: 2021-11-20T07:39:19.489606+0000 mon.smithi071 (mon.0) 566 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T07:39:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11-20T07:39:20.374583+0000 mon.smithi071 (mon.0) 567 : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:39:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11-20T07:39:20.376064+0000 2021-11-20T07:39:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 221 : cluster [DBG] 2021-11-20T07:39:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: pgmap v185: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:39:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:39:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: 20T07:39:20.379540 2021-11-20T07:39:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 568 : cluster [DBG] 2021-11-20T07:39:20.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:boot 2021-11-20T07:39:20.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:39:20.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: T07:39:20.379610+0000 mon.smithi071 ( 2021-11-20T07:39:20.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: mon.0) 569 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:20.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: audit 2021 2021-11-20T07:39:20.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: -11-20T07:39:20 2021-11-20T07:39:20.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: .379925+0000 mon.smithi071 (mon.0) 570 : audit 2021-11-20T07:39:20.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:20 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:39:22.467 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:22 smithi071 conmon[33687]: cluster 2021-11-20T07:39:21.384479+0000 mon.smithi071 (mon.0) 571 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:resolve 2021-11-20T07:39:22.467 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:22 smithi071 conmon[33687]: cluster 2021-11-20T07:39:21.384546+0000 mon.smithi071 (mon.0) 572 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:22.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:22 smithi112 conmon[36062]: cluster 2021-11-20T07:39:21.384479+0000 mon.smithi071 (mon.0) 571 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:resolve 2021-11-20T07:39:22.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:22 smithi112 conmon[36062]: cluster 2021-11-20T07:39:21.384546+0000 mon.smithi071 (mon.0) 572 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:23.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:23 smithi112 conmon[36062]: cluster 2021-11-20T07:39:22.376664+0000 mgr.smithi071.ynqmwn (mgr.14162) 222 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:39:23.992 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:23 smithi071 conmon[33687]: cluster 2021-11-20T07:39:22.376664+0000 mgr.smithi071.ynqmwn (mgr.14162) 222 : cluster 2021-11-20T07:39:23.992 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:23 smithi071 conmon[33687]: [DBG] pgmap v186: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:39:25.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:25 smithi112 conmon[36062]: cluster 2021-11-20T07:39:24.377086+0000 mgr.smithi071.ynqmwn (mgr.14162) 223 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:25 smithi112 conmon[36062]: cluster 2021-11-20T07:39:24.399928+0000 mon.smithi071 (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:standby 2021-11-20T07:39:25.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:25 smithi112 conmon[36062]: cluster 2021-11-20T07:39:24.399989+0000 mon.smithi071 (mon.0) 574 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:25 smithi071 conmon[33687]: cluster 2021-11-20T07:39:24.377086+0000 mgr.smithi071.ynqmwn (mgr.14162) 223 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:25 smithi071 conmon[33687]: cluster 2021-11-20T07:39:24.399928+0000 mon.smithi071 (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:standby 2021-11-20T07:39:25.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:25 smithi071 conmon[33687]: cluster 2021-11-20T07:39:24.399989+0000 mon.smithi071 (mon.0 2021-11-20T07:39:25.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:25 smithi071 conmon[33687]: ) 574 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:39:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:26 smithi071 conmon[33687]: cluster 2021-11-20T07:39:26.377721+0000 mgr.smithi071.ynqmwn (mgr.14162) 224 : cluster [DBG] 2021-11-20T07:39:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:26 smithi071 conmon[33687]: pgmap v188: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:26 smithi112 conmon[36062]: cluster 2021-11-20T07:39:26.377721+0000 mgr.smithi071.ynqmwn (mgr.14162) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:28.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:28 smithi112 conmon[36062]: cluster 2021-11-20T07:39:28.378361+0000 mgr.smithi071.ynqmwn (mgr.14162) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:28.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:28 smithi071 conmon[33687]: cluster 2021-11-20T07:39:28.378361+0000 mgr.smithi071.ynqmwn (mgr.14162) 225 : cluster 2021-11-20T07:39:28.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:28 smithi071 conmon[33687]: [DBG] pgmap v189: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:39:31.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:31 smithi112 conmon[36062]: cluster 2021-11-20T07:39:30.378791+0000 mgr.smithi071.ynqmwn (mgr.14162) 226 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:39:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:31 smithi071 conmon[33687]: cluster 2021-11-20T07:39:30.378791+0000 mgr.smithi071.ynqmwn (mgr.14162) 226 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:39:31.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:31.899+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:39:31.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:31.899+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:39:31.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:31.899+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:39:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:33 smithi112 conmon[36062]: cluster 2021-11-20T07:39:32.379443+0000 mgr.smithi071.ynqmwn (mgr.14162) 227 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:39:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:33 smithi071 conmon[33687]: cluster 2021-11-20T07:39:32.379443+0000 mgr.smithi071.ynqmwn (mgr.14162) 227 : cluster [DBG] 2021-11-20T07:39:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:33 smithi071 conmon[33687]: pgmap v191: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:39:34.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:34 smithi112 conmon[36062]: audit 2021-11-20T07:39:34.365504+0000 mon.smithi071 (mon.0) 575 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:39:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:34 smithi071 conmon[33687]: audit 2021-11-20T07:39:34.365504+0000 mon.smithi071 (mon.0) 575 : audit 2021-11-20T07:39:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:34 smithi071 conmon[33687]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:39:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster 2021-11-20T07:39:34.379867 2021-11-20T07:39:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 228 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:39:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: audit 2021-11-20T07:39:34.748181+0000 mon.smithi071 (mon.0) 576 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:39:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster 2021-11-20T07:39:35.377966+0000 mon.smithi071 (mon.0) 577 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.xtyomb as rank 1 with standby daemon mds.cephfs.smithi071.zaejme 2021-11-20T07:39:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster 2021-11-20T07:39:35.378011+0000 mon.smithi071 (mon.0) 578 : cluster [INF] MDS daemon mds.cephfs.smithi071.xtyomb is removed because it is dead or otherwise unavailable. 2021-11-20T07:39:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:39:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: -11-20T07:39:35.381986 2021-11-20T07:39:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 579 2021-11-20T07:39:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T07:39:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:39:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: 11-20T07:39:35.385227 2021-11-20T07:39:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 580 : 2021-11-20T07:39:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:35 smithi112 conmon[36062]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:replay} 1 up:standby 2021-11-20T07:39:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster 2021-11-20T07:39:34.379867+0000 mgr.smithi071.ynqmwn 2021-11-20T07:39:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: (mgr.14162) 228 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:39:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: audit 2021-11-20T07:39:34.748181+0000 mon.smithi071 (mon.0) 576 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:39:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster 2021-11-20T07:39:35.377966+0000 mon.smithi071 (mon.0) 577 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.xtyomb as rank 1 with standby daemon mds.cephfs.smithi071.zaejme 2021-11-20T07:39:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster 2021-11-20T07:39:35.378011+0000 mon.smithi071 (mon.0) 578 : cluster [INF] MDS daemon mds.cephfs.smithi071.xtyomb is removed because it is dead or otherwise unavailable. 2021-11-20T07:39:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster 2021-11-20T07:39:35.381986+0000 mon.smithi071 (mon.0) 579 : 2021-11-20T07:39:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T07:39:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:35 smithi071 conmon[33687]: cluster 2021-11-20T07:39:35.385227+0000 mon.smithi071 (mon.0) 580 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:replay} 1 up:standby 2021-11-20T07:39:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: cluster 2021-11-20T07:39:36.380313+0000 mgr.smithi071.ynqmwn (mgr.14162) 229 : cluster 2021-11-20T07:39:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:39:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: cluster 2021-11-20T07:39:36.386595+0000 mon.smithi071 (mon.0) 581 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:resolve 2021-11-20T07:39:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: cluster 2021-11-20T07:39:36.386640+0000 2021-11-20T07:39:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: mon.smithi071 (mon.0) 582 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:boot 2021-11-20T07:39:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:39:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: -20T07:39:36.386694+0000 mon.smithi071 (mon.0) 583 : cluster [DBG] 2021-11-20T07:39:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:resolve} 2 up:standby 2021-11-20T07:39:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: audit 2021-11-20T07:39: 2021-11-20T07:39:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: 36.386858+0000 mon.smithi071 (mon.0) 584 : audit [DBG] 2021-11-20T07:39:37.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:37 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:39:37.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: cluster 2021-11-20T07:39:36.380313+0000 mgr.smithi071.ynqmwn (mgr.14162) 229 : cluster 2021-11-20T07:39:37.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:39:37.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: cluster 2021-11-20T07:39:36.386595+0000 mon.smithi071 (mon.0) 581 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:resolve 2021-11-20T07:39:37.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: cluster 2021-11-20T07:39:36.386640+0000 mon.smithi071 (mon.0) 582 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:boot 2021-11-20T07:39:37.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: cluster 2021-11-20T07:39:36.386694+0000 mon.smithi071 (mon.0) 583 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:resolve} 2 up:standby 2021-11-20T07:39:37.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:37 smithi071 conmon[33687]: audit 2021-11-20T07:39:36.386858+0000 mon.smithi071 (mon.0) 584 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:39:38.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: cluster 2021-11-20T07:39:37.392858+0000 mon.smithi071 (mon.0 2021-11-20T07:39:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: ) 585 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:reconnect 2021-11-20T07:39:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: cluster 2021-11-20T07:39:37.392900+0000 mon.smithi071 (mon.0) 586 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:reconnect 2021-11-20T07:39:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: cluster 2021-11-20T07:39:37.392958+0000 mon.smithi071 (mon.0) 587 : cluster 2021-11-20T07:39:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi071.zaejme=up:reconnect} 2 up:standby 2021-11-20T07:39:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: cluster 2021-11-20T07:39:37.417019+0000 2021-11-20T07:39:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: mds.cephfs.smithi071.zaejme (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249996 2021-11-20T07:39:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:39:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:38 smithi112 conmon[36062]: T07:39:37.417318+0000 mds.cephfs.smithi112.iqrmps (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:39:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: cluster 2021-11-20T07:39:37.392858+0000 mon.smithi071 (mon.0) 585 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:reconnect 2021-11-20T07:39:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: cluster 2021-11-20T07:39:37.392900+0000 mon.smithi071 (mon.0) 586 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:reconnect 2021-11-20T07:39:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: cluster 2021-11-20T07:39:37.392958+0000 mon.smithi071 (mon.0) 587 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi071.zaejme=up:reconnect} 2 up:standby 2021-11-20T07:39:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:39:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: 20T07:39:37.417019+0000 mds.cephfs.smithi071.zaejme (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249996 2021-11-20T07:39:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: cluster 2021-11-20T07:39:37. 2021-11-20T07:39:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:38 smithi071 conmon[33687]: 417318+0000 mds.cephfs.smithi112.iqrmps (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:39:39.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: cluster 2021-11-20T07:39:38.380776+0000 mgr.smithi071.ynqmwn 2021-11-20T07:39:39.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: (mgr.14162) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:39:39.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: cluster 2021-11-20T07:39:38.398968+0000 mon.smithi071 (mon.0) 588 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:rejoin 2021-11-20T07:39:39.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: cluster 2021-11-20T07:39:38.399026+0000 mon.smithi071 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:rejoin 2021-11-20T07:39:39.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: cluster 2021-11-20T07:39:38.399092+0000 mon.smithi071 (mon.0) 590 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi071.zaejme=up:rejoin} 2 up:standby 2021-11-20T07:39:39.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:39 smithi112 conmon[36062]: cluster 2021-11-20T07:39:38.400523+0000 mon.smithi071 (mon.0) 591 : cluster [INF] daemon mds.cephfs.smithi071.zaejme is now active in filesystem cephfs as rank 1 2021-11-20T07:39:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: cluster 2021-11-20T07:39:38.380776+0000 mgr.smithi071.ynqmwn (mgr.14162) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 4.1 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:39:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:39:39.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: 39:38.398968+0000 mon.smithi071 (mon.0) 588 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:rejoin 2021-11-20T07:39:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: cluster 2021-11-20T07:39:38.399026+0000 mon.smithi071 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:rejoin 2021-11-20T07:39:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: cluster 2021-11-20T07:39:38.399092+0000 mon.smithi071 (mon 2021-11-20T07:39:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: .0) 590 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi071.zaejme=up:rejoin} 2 up:standby 2021-11-20T07:39:39.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:39:39.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: -11-20T07:39:38 2021-11-20T07:39:39.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: .400523+0000 mon.smithi071 (mon.0) 591 : cluster 2021-11-20T07:39:39.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:39 smithi071 conmon[33687]: [INF] daemon mds.cephfs.smithi071.zaejme is now active in filesystem cephfs as rank 1 2021-11-20T07:39:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: cluster 2021-11-20T07:39:39.405768+0000 mon.smithi071 (mon.0) 2021-11-20T07:39:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: 592 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:active 2021-11-20T07:39:40.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: cluster 2021-11-20T07:39:39.405819+0000 mon.smithi071 (mon.0) 593 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:clientreplay 2021-11-20T07:39:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:39:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: 20T07:39:39.405888+0000 mon.smithi071 (mon.0) 594 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: cluster 2021-11-20T07:39:40 2021-11-20T07:39:40.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: .381280+0000 mgr.smithi071.ynqmwn (mgr.14162) 231 : cluster 2021-11-20T07:39:40.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:40 smithi071 conmon[33687]: [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.2 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:39:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: cluster 2021-11-20T07:39:39.405768+0000 mon.smithi071 (mon.0) 592 : cluster [DBG] mds.? [v2:172.21.15.71:6828/124280792,v1:172.21.15.71:6829/124280792] up:active 2021-11-20T07:39:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: cluster 2021-11-20T07:39:39.405819+0000 mon.smithi071 (mon.0) 593 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3800926744,v1:172.21.15.112:6827/3800926744] up:clientreplay 2021-11-20T07:39:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: cluster 2021-11-20T07:39:39.405888+0000 mon.smithi071 (mon.0) 2021-11-20T07:39:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: 594 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: cluster 2021-11-20T07:39:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: 2021-11-20T07:39:40.381280+0000 mgr.smithi071.ynqmwn 2021-11-20T07:39:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:40 smithi112 conmon[36062]: (mgr.14162) 231 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.2 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:39:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:41 smithi071 conmon[33687]: cluster 2021-11-20T07:39:40.412495+0000 mon.smithi071 (mon.0) 595 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:standby 2021-11-20T07:39:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:41 smithi071 conmon[33687]: cluster 2021-11-20T07:39: 2021-11-20T07:39:41.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:41 smithi071 conmon[33687]: 40.412560+0000 mon.smithi071 (mon.0) 596 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:41 smithi112 conmon[36062]: cluster 2021-11-20T07:39:40.412495+0000 mon.smithi071 (mon.0) 595 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:standby 2021-11-20T07:39:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:41 smithi112 conmon[36062]: cluster 2021-11-20T07:39:40.412560+0000 mon.smithi071 (mon.0) 596 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:42 smithi112 conmon[36062]: cluster 2021-11-20T07:39:42.381934+0000 mgr.smithi071.ynqmwn (mgr.14162) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:39:42.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:42 smithi071 conmon[33687]: cluster 2021-11-20T07:39:42.381934+0000 mgr.smithi071.ynqmwn (mgr.14162) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:39:44.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:44 smithi112 conmon[36062]: cluster 2021-11-20T07:39:44.382521+0000 mgr.smithi071.ynqmwn (mgr.14162) 233 : cluster [DBG] 2021-11-20T07:39:44.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:44 smithi112 conmon[36062]: pgmap v198: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:39:44.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:44 smithi071 conmon[33687]: cluster 2021-11-20T07:39:44.382521+0000 mgr.smithi071.ynqmwn (mgr.14162) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:39:46.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:46 smithi112 conmon[36062]: cluster 2021-11-20T07:39:46.382910+0000 mgr.smithi071.ynqmwn (mgr.14162) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 372 B/s wr, 1 op/s 2021-11-20T07:39:46.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:46 smithi112 conmon[36062]: 2021-11-20T07:39:46.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:46 smithi071 conmon[33687]: cluster 2021-11-20T07:39:46.382910+0000 mgr.smithi071.ynqmwn (mgr.14162) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 372 B/s wr, 1 op/s 2021-11-20T07:39:46.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:46.902+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:39:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:46.903+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:39:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:46.903+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:39:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:46.903+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:39:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:39:46.903+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:39:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:48 smithi112 conmon[36062]: cluster 2021-11-20T07:39:48.383575+0000 mgr.smithi071.ynqmwn (mgr.14162) 235 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:39:48.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:48 smithi071 conmon[33687]: cluster 2021-11-20T07:39:48.383575+0000 mgr.smithi071.ynqmwn (mgr.14162) 235 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:39:51.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:51 smithi112 conmon[36062]: cluster 2021-11-20T07:39:50.384007+0000 mgr.smithi071.ynqmwn (mgr.14162) 236 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:39:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:51 smithi071 conmon[33687]: cluster 2021-11-20T07:39:50.384007+0000 mgr.smithi071.ynqmwn (mgr.14162) 236 : cluster 2021-11-20T07:39:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:51 smithi071 conmon[33687]: [DBG] pgmap v201: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:39:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:53 smithi112 conmon[36062]: cluster 2021-11-20T07:39:52.384537+0000 mgr.smithi071.ynqmwn (mgr.14162) 237 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:39:53.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:53 smithi071 conmon[33687]: cluster 2021-11-20T07:39:52.384537+0000 mgr.smithi071.ynqmwn (mgr.14162) 237 : 2021-11-20T07:39:53.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:53 smithi071 conmon[33687]: cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:39:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:54 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.278506+0000 mon.smithi071 (mon.0) 597 : cluster [INF] daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:39:54.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:54 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.284735+0000 mon.smithi071 (mon.0) 598 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T07:39:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:54 smithi071 conmon[33687]: cluster 2021-11-20T07:39:54.278506+0000 mon.smithi071 (mon.0) 597 : cluster [INF] 2021-11-20T07:39:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:54 smithi071 conmon[33687]: daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:39:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:54 smithi071 conmon[33687]: cluster 2021-11-20T07:39:54.284735+0000 mon.smithi071 (mon.0) 598 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T07:39:55.515 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: cluster 2021-11-20T07:39:54.385059+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:39:55.516 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: ) 238 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:39:55.516 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: cluster 2021-11-20T07:39:54.402279+0000 mon.smithi071 (mon.0) 599 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:boot 2021-11-20T07:39:55.516 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: cluster 2021-11-20T07:39:54.402333+0000 mon.smithi071 ( 2021-11-20T07:39:55.517 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: mon.0) 600 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 2021-11-20T07:39:55.517 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:39:55.517 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: 39:54.402897+0000 mon.smithi071 (mon.0) 601 2021-11-20T07:39:55.517 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.zaejme=up:active} 3 up:standby, 1 failed 2021-11-20T07:39:55.517 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:39:55.518 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: 39:54.403277+0000 mon.smithi071 (mon.0) 2021-11-20T07:39:55.518 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: 602 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:39:55.518 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:39:55.518 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: 07:39:54.405763+0000 mon.smithi071 (mon. 2021-11-20T07:39:55.518 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:55 smithi071 conmon[33687]: 0) 603 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:replay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.385059+0000 mgr.smithi071.ynqmwn (mgr.14162) 238 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:39:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.402279+0000 mon.smithi071 (mon.0) 599 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:boot 2021-11-20T07:39:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.402333+0000 mon.smithi071 (mon.0) 600 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 2021-11-20T07:39:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: cluster 2021-11-20T07:39:54.402897+0000 mon.smithi071 (mon.0) 601 : cluster 2021-11-20T07:39:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.zaejme=up:active} 3 up:standby, 1 failed 2021-11-20T07:39:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: audit 2021-11-20T07:39 2021-11-20T07:39:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: :54.403277+0000 mon.smithi071 (mon.0) 602 : audit 2021-11-20T07:39:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:39:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:39:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: 39:54.405763+0000 mon.smithi071 (mon.0) 603 2021-11-20T07:39:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:55 smithi112 conmon[36062]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:replay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:56 smithi112 conmon[36062]: cluster 2021-11-20T07:39:55.408161+0000 mon.smithi071 (mon.0) 604 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:resolve 2021-11-20T07:39:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:56 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:39:56.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:56 smithi112 conmon[36062]: :39:55.408239+0000 mon.smithi071 (mon.0) 605 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:56.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:56 smithi112 conmon[36062]: cluster 2021-11-20T07:39:56.385659+0000 mgr.smithi071.ynqmwn (mgr.14162) 239 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T07:39:56.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:56 smithi071 conmon[33687]: cluster 2021-11-20T07:39:55.408161+0000 mon.smithi071 (mon.0) 604 : cluster [DBG] 2021-11-20T07:39:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:56 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:resolve 2021-11-20T07:39:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:56 smithi071 conmon[33687]: cluster 2021-11-20T07:39:55.408239+0000 mon.smithi071 (mon.0) 605 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:56.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:56 smithi071 conmon[33687]: cluster 2021-11-20T07:39:56.385659+0000 mgr.smithi071.ynqmwn (mgr.14162) 239 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T07:39:59.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:59 smithi071 conmon[33687]: cluster 2021-11-20T07:39:58.332733+0000 mon.smithi071 (mon.0) 606 : cluster [DBG] 2021-11-20T07:39:59.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:59 smithi071 conmon[33687]: mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:standby 2021-11-20T07:39:59.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:59 smithi071 conmon[33687]: cluster 2021-11-20T07:39:58.332807+0000 mon.smithi071 (mon.0) 607 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:59.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:59 smithi071 conmon[33687]: cluster 2021-11-20T07:39:58.386374+0000 mgr.smithi071.ynqmwn (mgr.14162) 240 : cluster [DBG] 2021-11-20T07:39:59.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:39:59 smithi071 conmon[33687]: pgmap v206: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:39:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:59 smithi112 conmon[36062]: cluster 2021-11-20T07:39:58.332733+0000 mon.smithi071 (mon.0) 606 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:standby 2021-11-20T07:39:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:59 smithi112 conmon[36062]: cluster 2021-11-20T07:39:58.332807+0000 mon.smithi071 (mon.0) 607 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:39:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:39:59 smithi112 conmon[36062]: cluster 2021-11-20T07:39:58.386374+0000 mgr.smithi071.ynqmwn (mgr.14162) 240 : cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:40:00.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:00 smithi112 conmon[36062]: cluster 2021-11-20T07:40:00.000162+0000 mon.smithi071 (mon.0) 608 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded 2021-11-20T07:40:00.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:00 smithi112 conmon[36062]: cluster 2021-11-20T07:40:00.000249+0000 2021-11-20T07:40:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:00 smithi112 conmon[36062]: mon.smithi071 (mon.0) 609 : cluster [WRN] [WRN] FS_DEGRADED: 1 filesystem is degraded 2021-11-20T07:40:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:00 smithi112 conmon[36062]: cluster 2021-11-20T07:40:00.000275+0000 mon.smithi071 (mon.0) 610 : cluster [WRN] fs cephfs is degraded 2021-11-20T07:40:00.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:00 smithi071 conmon[33687]: cluster 2021-11-20T07:40:00.000162+0000 mon.smithi071 (mon.0) 608 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded 2021-11-20T07:40:00.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:00 smithi071 conmon[33687]: cluster 2021-11-20T07:40:00.000249+0000 mon.smithi071 (mon.0) 609 : cluster [WRN] [WRN] FS_DEGRADED: 1 filesystem is degraded 2021-11-20T07:40:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:00 smithi071 conmon[33687]: cluster 2021-11-20T07:40:00.000275+0000 mon.smithi071 (mon. 2021-11-20T07:40:00.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:00 smithi071 conmon[33687]: 0) 610 : cluster [WRN] fs cephfs is degraded 2021-11-20T07:40:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:01 smithi112 conmon[36062]: cluster 2021-11-20T07:40:00.386775+0000 mgr.smithi071.ynqmwn (mgr.14162) 241 : cluster [DBG] 2021-11-20T07:40:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:01 smithi112 conmon[36062]: pgmap v207: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:01 smithi071 conmon[33687]: cluster 2021-11-20T07 2021-11-20T07:40:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:01 smithi071 conmon[33687]: :40:00.386775+0000 mgr.smithi071.ynqmwn (mgr.14162) 241 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:01.907 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:01.905+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:01.907 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:01.905+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:40:01.908 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:01.905+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:40:01.908 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:01.905+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:40:01.908 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:01.905+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:40:03.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:03 smithi071 conmon[33687]: cluster 2021-11-20T07:40:02.387461+0000 mgr.smithi071.ynqmwn (mgr.14162) 242 : cluster [DBG] 2021-11-20T07:40:03.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:03 smithi071 conmon[33687]: pgmap v208: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:03 smithi112 conmon[36062]: cluster 2021-11-20T07:40:02.387461+0000 mgr.smithi071.ynqmwn (mgr.14162) 242 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:03 smithi112 conmon[36062]: 2021-11-20T07:40:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:05 smithi112 conmon[36062]: cluster 2021-11-20T07:40:04.387903+0000 mgr.smithi071.ynqmwn (mgr.14162) 243 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:05 smithi071 conmon[33687]: cluster 2021-11-20T07:40:04.387903+0000 mgr.smithi071.ynqmwn (mgr.14162) 243 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:06 smithi112 conmon[36062]: cluster 2021-11-20T07:40:06.324791+0000 mon.smithi071 (mon.0) 611 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:40:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:06 smithi112 conmon[36062]: cluster 2021-11-20T07:40:06.328916+0000 mon.smithi071 (mon.0) 612 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T07:40:06.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:06 smithi071 conmon[33687]: cluster 2021-11-20T07:40:06.324791+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:06.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:06 smithi071 conmon[33687]: 611 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:40:06.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:06 smithi071 conmon[33687]: cluster 2021-11-20T07:40:06.328916+0000 mon.smithi071 (mon.0) 612 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T07:40:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster 2021-11-20T07:40:06.388318+0000 mgr.smithi071.ynqmwn (mgr.14162) 244 : 2021-11-20T07:40:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:40:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster 2021-11-20T07:40:06.401400+0000 mon.smithi071 (mon.0) 613 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:boot 2021-11-20T07:40:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster 2021-11-20T07:40:06.401439+0000 mon.smithi071 (mon.0) 614 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:40:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster 2021-11-20T07:40:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: 2021-11-20T07:40:06.401681+0000 mon.smithi071 (mon.0 2021-11-20T07:40:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: ) 615 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi071.xtyomb=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:40:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:40:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: T07:40:06.401783+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: 616 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:40:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:40:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: T07:40:06.405495+0000 mon.smithi071 (mon.0) 617 : cluster 2021-11-20T07:40:07.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:07 smithi112 conmon[36062]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2 up:standby 2021-11-20T07:40:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: cluster 2021-11-20T07:40:06.388318+0000 mgr.smithi071.ynqmwn (mgr.14162) 244 : cluster 2021-11-20T07:40:07.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: [DBG] pgmap v211: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:40:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: cluster 2021-11-20T07:40:06.401400+0000 mon.smithi071 (mon.0) 613 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:boot 2021-11-20T07:40:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:40:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: -11-20T07:40:06.401439+0000 mon.smithi071 (mon.0) 614 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:40:07.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: cluster 2021-11-20T07:40:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: 2021-11-20T07:40:06.401681+0000 mon.smithi071 (mon.0) 615 2021-11-20T07:40:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi071.xtyomb=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:40:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: audit 2021-11-20T07:40:06. 2021-11-20T07:40:07.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: 401783+0000 mon.smithi071 (mon.0) 616 : 2021-11-20T07:40:07.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:40:07.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:40:07.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: 11-20T07:40:06. 2021-11-20T07:40:07.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: 405495+0000 mon.smithi071 (mon.0) 617 : cluster 2021-11-20T07:40:07.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:07 smithi071 conmon[33687]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2 up:standby 2021-11-20T07:40:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:08 smithi112 conmon[36062]: cluster 2021-11-20T07:40:07.408746+0000 mon.smithi071 (mon.0) 618 2021-11-20T07:40:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:08 smithi112 conmon[36062]: : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:resolve 2021-11-20T07:40:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:08 smithi112 conmon[36062]: cluster 2021-11-20T07:40:07.408820+0000 mon.smithi071 (mon.0) 619 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2 up:standby 2021-11-20T07:40:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:08 smithi112 conmon[36062]: cluster 2021-11-20T07:40:08.388767+0000 mgr.smithi071.ynqmwn (mgr.14162) 245 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:40:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:08 smithi071 conmon[33687]: cluster 2021-11-20T07:40:07.408746+0000 mon.smithi071 (mon.0) 618 2021-11-20T07:40:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:08 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:resolve 2021-11-20T07:40:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:08 smithi071 conmon[33687]: cluster 2021-11-20T07:40:07.408820+0000 mon.smithi071 (mon.0) 619 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2 up:standby 2021-11-20T07:40:08.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:08 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:40:08.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:08 smithi071 conmon[33687]: 20T07:40:08.388767+0000 mgr.smithi071.ynqmwn (mgr.14162) 245 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:40:09.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: cluster 2021-11-20T07:40:08.417697+0000 mon.smithi071 (mon.0) 620 : cluster [DBG] 2021-11-20T07:40:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:reconnect 2021-11-20T07:40:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: cluster 2021-11-20T07:40:08.417763+0000 mon.smithi071 (mon.0) 621 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:reconnect 2021-11-20T07:40:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: cluster 2021-11-20T07:40:08.417818+0000 mon.smithi071 (mon.0) 622 : cluster [DBG] 2021-11-20T07:40:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2 up:standby 2021-11-20T07:40:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: cluster 2021-11-20T07:40: 2021-11-20T07:40:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: 08.442365+0000 mds.cephfs.smithi071.xtyomb (mds.0) 1 : cluster [DBG] 2021-11-20T07:40:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269995 2021-11-20T07:40:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: cluster 2021-11-20T07:40:08.442674+0000 2021-11-20T07:40:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:09 smithi112 conmon[36062]: mds.cephfs.smithi112.bxcrgi (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259993 2021-11-20T07:40:09.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: cluster 2021-11-20T07:40:08.417697+0000 mon.smithi071 (mon.0) 620 : cluster [DBG] 2021-11-20T07:40:09.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:reconnect 2021-11-20T07:40:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: cluster 2021-11-20T07:40:08.417763+0000 mon.smithi071 (mon.0) 621 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:reconnect 2021-11-20T07:40:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: cluster 2021-11-20T07:40:08.417818+0000 mon.smithi071 (mon 2021-11-20T07:40:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: .0) 622 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2 up:standby 2021-11-20T07:40:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:40:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: 07:40 2021-11-20T07:40:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: :08.442365+0000 mds.cephfs.smithi071.xtyomb (mds.0) 1 : cluster [DBG] 2021-11-20T07:40:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269995 2021-11-20T07:40:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:40:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:09 smithi071 conmon[33687]: 07:40:08.442674+0000 mds.cephfs.smithi112.bxcrgi (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259993 2021-11-20T07:40:10.567 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: cluster 2021-11-20T07:40:09.425588+0000 mon.smithi071 (mon.0) 623 : cluster [DBG] 2021-11-20T07:40:10.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:rejoin 2021-11-20T07:40:10.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: cluster 2021-11-20T07:40:09.425656+0000 mon.smithi071 (mon.0) 624 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:rejoin 2021-11-20T07:40:10.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:40:10.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: 40:09.425763+0000 mon.smithi071 (mon.0) 625 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2 up:standby 2021-11-20T07:40:10.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: cluster 2021-11-20T07:40 2021-11-20T07:40:10.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: :09.427734+0000 mon.smithi071 (mon.0) 626 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:40:10.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: cluster 2021-11-20T07:40:10.389169+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:40:10.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:10 smithi071 conmon[33687]: ) 246 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T07:40:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster 2021-11-20T07:40:09.425588+0000 mon.smithi071 (mon.0) 623 : 2021-11-20T07:40:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:rejoin 2021-11-20T07:40:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster 2021-11-20T07:40:09.425656+0000 mon.smithi071 (mon.0) 624 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:rejoin 2021-11-20T07:40:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster 2021-11-20T07:40:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: 2021-11-20T07:40:09.425763+0000 mon.smithi071 (mon.0) 625 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2 up:standby 2021-11-20T07:40:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster 2021-11-20T07:40: 2021-11-20T07:40:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: 09.427734+0000 mon.smithi071 (mon.0) 626 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:40:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: 2021-11-20T07:40:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: cluster 2021-11-20T07:40:10 2021-11-20T07:40:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: .389169+0000 mgr.smithi071.ynqmwn (mgr.14162) 246 : cluster [DBG] 2021-11-20T07:40:10.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:10 smithi112 conmon[36062]: pgmap v213: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T07:40:11.489 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:11 smithi071 conmon[33687]: cluster 2021-11-20T07:40:10.428372+0000 mon.smithi071 (mon.0) 627 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:clientreplay 2021-11-20T07:40:11.489 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:11 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:40:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:11 smithi071 conmon[33687]: -11-20T07:40:10.428411+0000 mon.smithi071 (mon.0) 628 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:active 2021-11-20T07:40:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:11 smithi071 conmon[33687]: cluster 2021-11-20T07:40:10.428445+0000 mon.smithi071 (mon.0) 629 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:standby 2021-11-20T07:40:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:11 smithi071 conmon[33687]: cluster 2021-11-20T07:40:10.428486+0000 mon.smithi071 (mon.0) 630 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: cluster 2021-11-20T07:40:10.428372+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: 627 : cluster [DBG] mds.? [v2:172.21.15.71:6826/2092546221,v1:172.21.15.71:6827/2092546221] up:clientreplay 2021-11-20T07:40:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: cluster 2021-11-20T07:40:10.428411+0000 mon.smithi071 (mon.0) 628 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1014359088,v1:172.21.15.112:6825/1014359088] up:active 2021-11-20T07:40:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: cluster 2021-11-20T07:40:10.428445+0000 mon.smithi071 (mon. 2021-11-20T07:40:11.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: 0) 629 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:standby 2021-11-20T07:40:11.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:40:11.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: -20T07:40:10.428486+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:11.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:11 smithi112 conmon[36062]: 630 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:12 smithi112 conmon[36062]: cluster 2021-11-20T07:40:12.389718+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:40:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:12 smithi112 conmon[36062]: ) 247 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 4.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:40:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:12 smithi071 conmon[33687]: cluster 2021-11-20T07:40:12.389718+0000 mgr.smithi071.ynqmwn (mgr.14162) 247 : cluster [DBG] 2021-11-20T07:40:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:12 smithi071 conmon[33687]: pgmap v214: 65 pgs: 65 active+clean; 4.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:40:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:14 smithi112 conmon[36062]: cluster 2021-11-20T07:40:14.390206+0000 mgr.smithi071.ynqmwn (mgr.14162) 248 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:40:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:14 smithi071 conmon[33687]: cluster 2021-11-20T07:40:14.390206+0000 mgr.smithi071.ynqmwn (mgr.14162) 248 : cluster [DBG] 2021-11-20T07:40:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:14 smithi071 conmon[33687]: pgmap v215: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:40:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:16 smithi112 conmon[36062]: cluster 2021-11-20T07:40:16.390597+0000 mgr.smithi071.ynqmwn (mgr.14162) 249 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 407 B/s wr, 1 op/s 2021-11-20T07:40:16.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:16 smithi071 conmon[33687]: cluster 2021-11-20T07:40:16.390597+0000 mgr.smithi071.ynqmwn (mgr.14162) 249 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 407 B/s wr, 1 op/s 2021-11-20T07:40:16.915 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:16.913+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:6826/657608204 2021-11-20T07:40:16.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:16.913+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:40:16.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:16.913+0000 7f2998ff9700 1 client.14546 ms_handle_refused on v2:172.21.15.71:6826/657608204 2021-11-20T07:40:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:18 smithi112 conmon[36062]: cluster 2021-11-20T07:40:18.391252+0000 mgr.smithi071.ynqmwn (mgr.14162) 250 : cluster [DBG] 2021-11-20T07:40:18.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:18 smithi112 conmon[36062]: pgmap v217: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:40:18.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:18 smithi071 conmon[33687]: cluster 2021-11-20T07:40:18.391252+0000 mgr.smithi071.ynqmwn (mgr.14162) 250 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:40:18.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:18 smithi071 conmon[33687]: 2021-11-20T07:40:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:19 smithi071 conmon[33687]: audit 2021-11-20T07:40:18.936549+0000 mon.smithi071 (mon.0) 631 : 2021-11-20T07:40:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:19 smithi071 conmon[33687]: audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:40:19.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:19 smithi071 conmon[33687]: audit 2021-11-20T07:40:18.936978+0000 mon.smithi071 (mon.0) 632 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:40:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:19 smithi112 conmon[36062]: audit 2021-11-20T07:40:18.936549+0000 mon.smithi071 (mon.0) 631 2021-11-20T07:40:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:19 smithi112 conmon[36062]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:40:19.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:19 smithi112 conmon[36062]: audit 2021-11-20T07:40:18.936978+0000 mon.smithi071 (mon.0) 632 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:40:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:21 smithi112 conmon[36062]: cluster 2021-11-20T07:40:20.391698+0000 mgr.smithi071.ynqmwn (mgr.14162) 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:40:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:21 smithi112 conmon[36062]: cluster 2021-11-20T07:40:21.354840+0000 mon.smithi071 (mon.0) 633 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:40:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:21 smithi112 conmon[36062]: cluster 2021-11-20T07:40:21.358551+0000 mon.smithi071 (mon.0) 634 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T07:40:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:21 smithi071 conmon[33687]: cluster 2021-11-20T07:40:20.391698+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:40:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:21 smithi071 conmon[33687]: mgr.14162) 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:40:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:21 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.354840+0000 mon.smithi071 (mon.0) 633 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:40:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:21 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.358551+0000 mon.smithi071 (mon.0) 634 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T07:40:22.522 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.392702+0000 mon.smithi071 (mon. 2021-11-20T07:40:22.523 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: 0) 635 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:boot 2021-11-20T07:40:22.524 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.392742+0000 mon.smithi071 (mon.0) 636 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:40:22.524 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.393032+0000 mon.smithi071 (mon.0) 637 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi112.bxcrgi=up:active} 3 up:standby, 1 failed 2021-11-20T07:40:22.524 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: audit 2021-11-20T07:40:21.393128+0000 mon.smithi071 (mon 2021-11-20T07:40:22.524 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: .0) 638 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:40:22.525 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: cluster 2021-11-20T07:40:21.400489+0000 mon.smithi071 ( 2021-11-20T07:40:22.525 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:22 smithi071 conmon[33687]: mon.0) 639 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:22.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: cluster 2021-11-20T07:40:21.392702+0000 mon.smithi071 (mon.0) 635 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:boot 2021-11-20T07:40:22.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: cluster 2021-11-20T07:40:21.392742+0000 mon.smithi071 (mon.0) 636 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:40:22.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: cluster 2021-11-20T07:40:21.393032+0000 mon.smithi071 (mon.0) 637 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi112.bxcrgi=up:active} 3 up:standby, 1 failed 2021-11-20T07:40:22.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: audit 2021-11-20T07:40:22.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: 2021-11-20T07:40:21.393128+0000 mon.smithi071 (mon.0) 638 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:40:22.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:40:22.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:22 smithi112 conmon[36062]: -20T07:40:21.400489+0000 mon.smithi071 (mon.0) 639 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:23.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:23 smithi112 conmon[36062]: cluster 2021-11-20T07:40:22.392130+0000 mgr.smithi071.ynqmwn (mgr.14162) 252 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 102 B/s wr, 0 op/s 2021-11-20T07:40:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:23 smithi112 conmon[36062]: cluster 2021-11-20T07:40:22.405501+0000 mon.smithi071 (mon.0) 640 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:resolve 2021-11-20T07:40:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:23 smithi112 conmon[36062]: cluster 2021-11-20T07:40:22.405575+0000 mon.smithi071 (mon.0) 641 : 2021-11-20T07:40:23.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:23 smithi112 conmon[36062]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:23.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:23 smithi071 conmon[33687]: cluster 2021-11-20T07:40:22.392130+0000 mgr.smithi071.ynqmwn (mgr.14162) 252 : cluster 2021-11-20T07:40:23.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:23 smithi071 conmon[33687]: [DBG] pgmap v220: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 102 B/s wr, 0 op/s 2021-11-20T07:40:23.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:23 smithi071 conmon[33687]: cluster 2021-11-20T07:40:22.405501+0000 mon.smithi071 (mon.0) 640 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:resolve 2021-11-20T07:40:23.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:23 smithi071 conmon[33687]: cluster 2021-11-20T07:40: 2021-11-20T07:40:23.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:23 smithi071 conmon[33687]: 22.405575+0000 mon.smithi071 (mon.0) 641 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:25.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:25 smithi112 conmon[36062]: cluster 2021-11-20T07:40:24.392694+0000 mgr.smithi071.ynqmwn (mgr.14162) 253 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:40:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:25 smithi071 conmon[33687]: cluster 2021-11-20T07:40:24.392694+0000 mgr.smithi071.ynqmwn (mgr.14162) 253 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:40:26.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:26 smithi071 conmon[33687]: cluster 2021-11-20T07:40:25.410021+0000 mon.smithi071 (mon.0) 642 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:standby 2021-11-20T07:40:26.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:26 smithi071 conmon[33687]: cluster 2021-11-20T07:40:25.410086+0000 mon.smithi071 (mon.0) 643 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:26.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:26 smithi071 conmon[33687]: cluster 2021-11-20T07:40:26.393109+0000 2021-11-20T07:40:26.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:26 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 254 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:26 smithi112 conmon[36062]: cluster 2021-11-20T07:40:25.410021+0000 mon.smithi071 (mon.0 2021-11-20T07:40:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:26 smithi112 conmon[36062]: ) 642 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:standby 2021-11-20T07:40:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:26 smithi112 conmon[36062]: cluster 2021-11-20T07:40:25.410086+0000 mon.smithi071 (mon.0) 643 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:40:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:26 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:40:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:26 smithi112 conmon[36062]: -11-20T07:40:26.393109+0000 mgr.smithi071.ynqmwn (mgr.14162) 254 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:28.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:28 smithi112 conmon[36062]: cluster 2021-11-20T07:40:28.393616+0000 mgr.smithi071.ynqmwn (mgr.14162) 255 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:28.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:28 smithi071 conmon[33687]: cluster 2021-11-20T07:40:28.393616+0000 mgr.smithi071.ynqmwn (mgr.14162) 255 : cluster [DBG] 2021-11-20T07:40:28.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:28 smithi071 conmon[33687]: pgmap v223: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:31.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:31 smithi112 conmon[36062]: cluster 2021-11-20T07:40:30.394056+0000 mgr.smithi071.ynqmwn (mgr.14162) 256 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:31 smithi071 conmon[33687]: cluster 2021-11-20T07:40:30.394056+0000 mgr.smithi071.ynqmwn (mgr.14162) 256 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:31.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:31.915+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:31.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:31.915+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T07:40:31.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:31.915+0000 7f29bd9a1700 1 -- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 msgr2=0x55d607849a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T07:40:31.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:31.915+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T07:40:31.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:31.915+0000 7f29bd9a1700 1 --2- 172.21.15.71:0/498837014 >> [v2:172.21.15.71:6826/657608204,v1:172.21.15.71:6827/657608204] conn(0x55d607847670 0x55d607849a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T07:40:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:33 smithi112 conmon[36062]: cluster 2021-11-20T07:40:32.394745+0000 mgr.smithi071.ynqmwn (mgr.14162) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:40:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:33 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:40:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:33 smithi112 conmon[36062]: :40:33.304886+0000 mon.smithi071 (mon.0) 644 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:40:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:33 smithi112 conmon[36062]: cluster 2021-11-20T07:40:33.312073+0000 mon.smithi071 (mon.0) 645 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T07:40:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:33 smithi071 conmon[33687]: cluster 2021-11-20T07:40:32.394745+0000 mgr.smithi071.ynqmwn (mgr.14162) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:40:33.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:33 smithi071 conmon[33687]: cluster 2021-11-20T07:40:33.304886+0000 mon.smithi071 (mon.0) 644 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:40:33.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:33 smithi071 conmon[33687]: cluster 2021-11-20T07:40:33.312073+0000 mon.smithi071 (mon.0) 645 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T07:40:34.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: cluster 2021-11-20T07:40:33.406480+0000 mon.smithi071 (mon.0) 646 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:boot 2021-11-20T07:40:34.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:40:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: :40:33.406564+0000 mon.smithi071 (mon.0) 647 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 1 2021-11-20T07:40:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: cluster 2021-11-20T07:40:33.406883+0000 mon.smithi071 (mon.0) 648 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi071.zaejme=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:40:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: audit 2021-11-20T07:40:33.407011+0000 mon.smithi071 (mon.0) 649 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:40:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: cluster 2021-11-20T07:40:33 2021-11-20T07:40:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:34 smithi112 conmon[36062]: .409639+0000 mon.smithi071 (mon.0) 650 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 2 up:standby 2021-11-20T07:40:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: cluster 2021-11-20T07:40:33.406480+0000 mon.smithi071 (mon.0) 646 : cluster [DBG] 2021-11-20T07:40:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:boot 2021-11-20T07:40:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: cluster 2021-11-20T07:40:33.406564+0000 mon.smithi071 (mon.0) 647 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 1 2021-11-20T07:40:34.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: cluster 2021-11-20T07:40:33.406883 2021-11-20T07:40:34.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 648 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi071.zaejme=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:40:34.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: audit 2021-11-20T07:40:34.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: 2021-11-20T07:40:33 2021-11-20T07:40:34.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: .407011+0000 mon.smithi071 (mon.0) 649 2021-11-20T07:40:34.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:40:34.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: cluster 2021-11-20T07:40:34.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: 2021-11-20T07:40:33.409639+0000 2021-11-20T07:40:34.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: mon.smithi071 (mon.0) 650 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 2 up:standby 2021-11-20T07:40:34.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:34 smithi071 conmon[33687]: 2021-11-20T07:40:35.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: cluster 2021-11-20T07:40:34.395153+0000 mgr.smithi071.ynqmwn (mgr.14162) 258 : cluster 2021-11-20T07:40:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: [DBG] pgmap v227: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: cluster 2021-11-20T07:40:34.418222+0000 mon.smithi071 (mon.0) 651 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:resolve 2021-11-20T07:40:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: cluster 2021-11-20T07:40:34.418294+0000 mon.smithi071 (mon.0) 652 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:40:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: audit 2021-11-20T07:40:34.751557+0000 mon.smithi071 (mon.0) 653 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:40:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:35 smithi112 conmon[36062]: audit 2021-11-20T07:40:35.132415+0000 mon.smithi071 (mon.0) 654 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:40:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:35 smithi071 conmon[33687]: cluster 2021-11-20T07:40:34.395153+0000 mgr.smithi071.ynqmwn (mgr.14162) 258 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:35 smithi071 conmon[33687]: cluster 2021-11-20T07:40:34.418222+0000 mon.smithi071 (mon.0) 651 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:resolve 2021-11-20T07:40:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:35 smithi071 conmon[33687]: cluster 2021-11-20T07:40:34.418294+0000 mon.smithi071 (mon.0) 652 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:40:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:35 smithi071 conmon[33687]: audit 2021-11-20T07:40:34.751557+0000 mon.smithi071 (mon.0) 653 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:40:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:35 smithi071 conmon[33687]: audit 2021-11-20T07:40:35.132415+0000 mon.smithi071 (mon.0) 654 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:40:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-11-20T07:40:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:35.418503+0000 mon.smithi071 (mon.0) 655 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:reconnect 2021-11-20T07:40:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:35.418538+0000 mon.smithi071 (mon.0) 656 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:reconnect 2021-11-20T07:40:36.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:35.418602+0000 mon.smithi071 (mon.0) 657 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:40:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:35.442501+0000 mds.cephfs.smithi071.xtyomb (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:40:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:35.442531+0000 mds.cephfs.smithi071.zaejme (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:40:36.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:36 smithi112 conmon[36062]: cluster 2021-11-20T07:40:36.395595+0000 mgr.smithi071.ynqmwn (mgr.14162) 259 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:40:36.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07:40:35.418503+0000 mon.smithi071 2021-11-20T07:40:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: (mon.0) 655 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:reconnect 2021-11-20T07:40:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07:40:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: 2021-11-20T07:40 2021-11-20T07:40:36.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: :35.418538+0000 mon.smithi071 (mon.0) 656 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:reconnect 2021-11-20T07:40:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07:40:35.418602+0000 mon.smithi071 (mon.0) 657 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:40:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07:40:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: 2021-11-20T07:40:35.442501+0000 mds.cephfs.smithi071.xtyomb (mds.1) 1 2021-11-20T07:40:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:40:36.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:40:36.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: 40:35.442531+0000 mds.cephfs.smithi071.zaejme (mds.0) 1 : cluster 2021-11-20T07:40:36.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:40:36.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: cluster 2021-11-20T07:40:36.395595+0000 2021-11-20T07:40:36.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:36 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 259 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:40:37.383 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:37.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- 172.21.15.71:0/3756501304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f67a0 msgr2=0x7f3fc00f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:37.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/3756501304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f67a0 0x7f3fc00f6c00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f3fac004660 tx=0x7f3fac009b30).stop 2021-11-20T07:40:37.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- 172.21.15.71:0/3756501304 shutdown_connections 2021-11-20T07:40:37.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/3756501304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f67a0 0x7f3fc00f6c00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/3756501304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f5500 0x7f3fc00f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- 172.21.15.71:0/3756501304 >> 172.21.15.71:0/3756501304 conn(0x7f3fc00f0c40 msgr2=0x7f3fc00f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:37.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- 172.21.15.71:0/3756501304 shutdown_connections 2021-11-20T07:40:37.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- 172.21.15.71:0/3756501304 wait complete. 2021-11-20T07:40:37.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 Processor -- start 2021-11-20T07:40:37.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.517+0000 7f3fc58bf700 1 -- start start 2021-11-20T07:40:37.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:37.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f67a0 0x7f3fc0107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:37.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0107ae0 con 0x7f3fc00f5500 2021-11-20T07:40:37.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0107c20 con 0x7f3fc00f67a0 2021-11-20T07:40:37.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:37.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48448/0 (socket says 172.21.15.71:48448) 2021-11-20T07:40:37.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 -- 172.21.15.71:0/4011857011 learned_addr learned my addr 172.21.15.71:0/4011857011 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:37.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 -- 172.21.15.71:0/4011857011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f67a0 msgr2=0x7f3fc0107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:37.525 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fbffff700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f67a0 0x7f3fc0107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:37.525 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f67a0 0x7f3fc0107530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.526 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 -- 172.21.15.71:0/4011857011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac005040 con 0x7f3fc00f5500 2021-11-20T07:40:37.526 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc48bd700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3fb4002700 tx=0x7f3fb4009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:37.526 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb4009320 con 0x7f3fc00f5500 2021-11-20T07:40:37.526 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc01085c0 con 0x7f3fc00f5500 2021-11-20T07:40:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb400f040 con 0x7f3fc00f5500 2021-11-20T07:40:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb4014900 con 0x7f3fc00f5500 2021-11-20T07:40:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.518+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc0108b80 con 0x7f3fc00f5500 2021-11-20T07:40:37.527 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.519+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc0045ad0 con 0x7f3fc00f5500 2021-11-20T07:40:37.528 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.519+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3fb4009480 con 0x7f3fc00f5500 2021-11-20T07:40:37.529 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.520+0000 7f3fbd7fa700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 0x7f3fb005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:37.529 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.520+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3fb40764c0 con 0x7f3fc00f5500 2021-11-20T07:40:37.529 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.520+0000 7f3fbffff700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 0x7f3fb005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:37.529 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.520+0000 7f3fbffff700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 0x7f3fb005e7c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3fac009d60 tx=0x7f3fac009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:37.530 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.524+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fb4048ed0 con 0x7f3fc00f5500 2021-11-20T07:40:37.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: cluster 2021-11-20T07:40:36. 2021-11-20T07:40:37.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: 426310+0000 mon.smithi071 (mon.0) 658 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:rejoin 2021-11-20T07:40:37.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: cluster 2021-11-20T07:40:37.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: 2021-11-20T07:40:36.426358+0000 2021-11-20T07:40:37.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: mon.smithi071 (mon.0) 659 : cluster 2021-11-20T07:40:37.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:rejoin 2021-11-20T07:40:37.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:40:37.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: 11-20T07:40:36.426419+0000 mon.smithi071 2021-11-20T07:40:37.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: (mon.0) 660 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:40:37.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:40:37.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: 11-20T07:40:36.427423+0000 2021-11-20T07:40:37.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: mon.smithi071 (mon.0) 661 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 1 2021-11-20T07:40:37.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:37 smithi071 conmon[33687]: 2021-11-20T07:40:37.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.713+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc0109220 con 0x7f3fc00f5500 2021-11-20T07:40:37.716 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.714+0000 7f3fbd7fa700 1 -- 172.21.15.71:0/4011857011 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f3fb404c4f0 con 0x7f3fc00f5500 2021-11-20T07:40:37.717 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:37.717 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:37.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 msgr2=0x7f3fb005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:37.718 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 0x7f3fb005e7c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3fac009d60 tx=0x7f3fac009b30).stop 2021-11-20T07:40:37.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 msgr2=0x7f3fc0107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:37.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3fb4002700 tx=0x7f3fb4009d80).stop 2021-11-20T07:40:37.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 shutdown_connections 2021-11-20T07:40:37.719 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fb005c300 0x7f3fb005e7c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc00f5500 0x7f3fc0107040 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 --2- 172.21.15.71:0/4011857011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fc00f67a0 0x7f3fc0107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:37.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 >> 172.21.15.71:0/4011857011 conn(0x7f3fc00f0c40 msgr2=0x7f3fc005f780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:37.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 shutdown_connections 2021-11-20T07:40:37.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:37.716+0000 7f3fc58bf700 1 -- 172.21.15.71:0/4011857011 wait complete. 2021-11-20T07:40:37.721 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:37.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:37.730 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: cluster 2021-11-20T07:40:36.426310+0000 mon.smithi071 ( 2021-11-20T07:40:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: mon.0) 658 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:rejoin 2021-11-20T07:40:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: cluster 2021-11-20T07:40:36.426358+0000 mon.smithi071 (mon.0) 659 : cluster [DBG] 2021-11-20T07:40:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:rejoin 2021-11-20T07:40:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:40:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: 40:36.426419+0000 mon.smithi071 (mon.0) 660 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:40:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:40:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: 11-20T07:40:36.427423+0000 mon.smithi071 (mon.0) 661 : 2021-11-20T07:40:37.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:37 smithi112 conmon[36062]: cluster [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 1 2021-11-20T07:40:38.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:38.758 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: cluster 2021-11-20T07:40:37.429539+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:38.758 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: 662 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:clientreplay 2021-11-20T07:40:38.758 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: cluster 2021-11-20T07:40:37.429577+0000 mon.smithi071 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:active 2021-11-20T07:40:38.759 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: cluster 2021-11-20T07:40:38.759 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: 2021-11-20T07:40:37.429629+0000 mon.smithi071 (mon.0) 664 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:standby 2021-11-20T07:40:38.759 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:40:38.759 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: T07:40:37.429688+0000 mon.smithi071 (mon 2021-11-20T07:40:38.760 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: .0) 665 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:clientreplay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:38.760 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:40:38.760 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: -20T07:40:37.715343+0000 mon.smithi071 2021-11-20T07:40:38.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: (mon.0) 666 : audit [DBG] from='client.? 172.21.15.71:0/4011857011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:38.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: cluster 2021-11-20T07:40:38.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: 2021-11-20 2021-11-20T07:40:38.761 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: T07:40:38.396092+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:40:38.762 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:38 smithi071 conmon[33687]: mgr.14162) 260 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:40:38.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster 2021-11-20T07:40:37.429539+0000 mon.smithi071 (mon 2021-11-20T07:40:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: .0) 662 : cluster [DBG] mds.? [v2:172.21.15.71:6828/1481695887,v1:172.21.15.71:6829/1481695887] up:clientreplay 2021-11-20T07:40:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster 2021-11-20T07:40:37.429577+0000 mon.smithi071 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1128459258,v1:172.21.15.71:6827/1128459258] up:active 2021-11-20T07:40:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster 2021-11-20T07:40: 2021-11-20T07:40:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: 37.429629+0000 mon.smithi071 (mon.0) 664 : cluster [DBG] 2021-11-20T07:40:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:standby 2021-11-20T07:40:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:40:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: 40:37.429688+0000 mon.smithi071 (mon.0) 665 : 2021-11-20T07:40:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:clientreplay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:40:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: 07:40:37.715343+0000 mon.smithi071 (mon.0) 666 2021-11-20T07:40:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/4011857011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:38.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:40:38.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: 11-20T07:40:38.396092+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:40:38.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:38 smithi112 conmon[36062]: .14162) 260 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:40:38.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.852+0000 7fc02258f700 1 -- 172.21.15.71:0/239784248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f6820 msgr2=0x7fc01c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:38.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.852+0000 7fc02258f700 1 --2- 172.21.15.71:0/239784248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f6820 0x7fc01c0f6c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fc010004660 tx=0x7fc010009b30).stop 2021-11-20T07:40:38.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- 172.21.15.71:0/239784248 shutdown_connections 2021-11-20T07:40:38.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 --2- 172.21.15.71:0/239784248 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f6820 0x7fc01c0f6c80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:38.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 --2- 172.21.15.71:0/239784248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc01c0f5580 0x7fc01c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:38.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- 172.21.15.71:0/239784248 >> 172.21.15.71:0/239784248 conn(0x7fc01c0f0ce0 msgr2=0x7fc01c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:38.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- 172.21.15.71:0/239784248 shutdown_connections 2021-11-20T07:40:38.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- 172.21.15.71:0/239784248 wait complete. 2021-11-20T07:40:38.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 Processor -- start 2021-11-20T07:40:38.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- start start 2021-11-20T07:40:38.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:38.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc01c0f6820 0x7fc01c10bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:38.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c10c190 con 0x7fc01c0f5580 2021-11-20T07:40:38.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc02258f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c10c2d0 con 0x7fc01c0f6820 2021-11-20T07:40:38.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.853+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:38.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48468/0 (socket says 172.21.15.71:48468) 2021-11-20T07:40:38.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 -- 172.21.15.71:0/310235288 learned_addr learned my addr 172.21.15.71:0/310235288 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:38.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 -- 172.21.15.71:0/310235288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc01c0f6820 msgr2=0x7fc01c10bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:38.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc01c0f6820 0x7fc01c10bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:38.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 -- 172.21.15.71:0/310235288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010005040 con 0x7fc01c0f5580 2021-11-20T07:40:38.863 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc01bfff700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc00c0093f0 tx=0x7fc00c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:38.863 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00c009ea0 con 0x7fc01c0f5580 2021-11-20T07:40:38.863 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c10c5b0 con 0x7fc01c0f5580 2021-11-20T07:40:38.863 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc00c013af0 con 0x7fc01c0f5580 2021-11-20T07:40:38.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00c00dad0 con 0x7fc01c0f5580 2021-11-20T07:40:38.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.854+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c10cb70 con 0x7fc01c0f5580 2021-11-20T07:40:38.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.855+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c027a00 con 0x7fc01c0f5580 2021-11-20T07:40:38.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.855+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc00c01a060 con 0x7fc01c0f5580 2021-11-20T07:40:38.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.856+0000 7fc018ff9700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 0x7fc00405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:38.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.856+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc00c0761b0 con 0x7fc01c0f5580 2021-11-20T07:40:38.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.856+0000 7fc01b7fe700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 0x7fc00405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:38.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.856+0000 7fc01b7fe700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 0x7fc00405e760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc010009f50 tx=0x7fc010009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:38.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:38.859+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc00c048b60 con 0x7fc01c0f5580 2021-11-20T07:40:39.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.051+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc01c0fddb0 con 0x7fc01c0f5580 2021-11-20T07:40:39.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.051+0000 7fc018ff9700 1 -- 172.21.15.71:0/310235288 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7fc00c04c180 con 0x7fc01c0f5580 2021-11-20T07:40:39.054 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:39.054 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:39.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 msgr2=0x7fc00405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:39.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 0x7fc00405e760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc010009f50 tx=0x7fc010009b30).stop 2021-11-20T07:40:39.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 msgr2=0x7fc01c060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:39.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fc00c0093f0 tx=0x7fc00c006ec0).stop 2021-11-20T07:40:39.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 shutdown_connections 2021-11-20T07:40:39.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc00405c2a0 0x7fc00405e760 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:39.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc01c0f5580 0x7fc01c060510 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:39.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 --2- 172.21.15.71:0/310235288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc01c0f6820 0x7fc01c10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:39.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 >> 172.21.15.71:0/310235288 conn(0x7fc01c0f0ce0 msgr2=0x7fc01c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:39.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 shutdown_connections 2021-11-20T07:40:39.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:39.053+0000 7fc02258f700 1 -- 172.21.15.71:0/310235288 wait complete. 2021-11-20T07:40:39.059 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:39.068 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:39.068 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:39 smithi112 conmon[36062]: audit 2021-11-20T07:40:39.052763+0000 mon.smithi071 (mon.0) 667 : audit [DBG] from='client.? 172.21.15.71:0/310235288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:39 smithi071 conmon[33687]: audit 2021-11-20T07:40:39.052763+0000 mon.smithi071 (mon.0) 667 2021-11-20T07:40:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:39 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/310235288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:40.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:40.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 -- 172.21.15.71:0/537526545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 msgr2=0x7ff8c00f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:40.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 --2- 172.21.15.71:0/537526545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c00f6ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7ff8b0004660 tx=0x7ff8b0009b30).stop 2021-11-20T07:40:40.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 -- 172.21.15.71:0/537526545 shutdown_connections 2021-11-20T07:40:40.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 --2- 172.21.15.71:0/537526545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 0x7ff8c00f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 --2- 172.21.15.71:0/537526545 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c00f6ac0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 -- 172.21.15.71:0/537526545 >> 172.21.15.71:0/537526545 conn(0x7ff8c00f1da0 msgr2=0x7ff8c00f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 -- 172.21.15.71:0/537526545 shutdown_connections 2021-11-20T07:40:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 -- 172.21.15.71:0/537526545 wait complete. 2021-11-20T07:40:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.195+0000 7ff8c790b700 1 Processor -- start 2021-11-20T07:40:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 -- start start 2021-11-20T07:40:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 0x7ff8c01073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:40.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c01079f0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c0107b30 con 0x7ff8c00f7940 2021-11-20T07:40:40.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:40.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48480/0 (socket says 172.21.15.71:48480) 2021-11-20T07:40:40.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 -- 172.21.15.71:0/3628122293 learned_addr learned my addr 172.21.15.71:0/3628122293 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:40.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c4ea6700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 0x7ff8c01073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:40.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 -- 172.21.15.71:0/3628122293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 msgr2=0x7ff8c01073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:40.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 0x7ff8c01073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:40.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 -- 172.21.15.71:0/3628122293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0005040 con 0x7ff8c00f66a0 2021-11-20T07:40:40.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c56a7700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7ff8b0009f50 tx=0x7ff8b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:40.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b0018070 con 0x7ff8c00f66a0 2021-11-20T07:40:40.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8b0013410 con 0x7ff8c00f66a0 2021-11-20T07:40:40.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c010c570 con 0x7ff8c00f66a0 2021-11-20T07:40:40.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.196+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b001c620 con 0x7ff8c00f66a0 2021-11-20T07:40:40.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.197+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c010cad0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.197+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c0027a00 con 0x7ff8c00f66a0 2021-11-20T07:40:40.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.198+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff8b001d420 con 0x7ff8c00f66a0 2021-11-20T07:40:40.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.198+0000 7ff8b5ffb700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 0x7ff8ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:40.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.198+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff8b00780e0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.198+0000 7ff8c4ea6700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 0x7ff8ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:40.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.199+0000 7ff8c4ea6700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 0x7ff8ac05e760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff8bc00b790 tx=0x7ff8bc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:40.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.203+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8b004a9d0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.393+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8c00feeb0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.394+0000 7ff8b5ffb700 1 -- 172.21.15.71:0/3628122293 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7ff8b004dff0 con 0x7ff8c00f66a0 2021-11-20T07:40:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:40.397 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:40.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 msgr2=0x7ff8ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:40.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 0x7ff8ac05e760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff8bc00b790 tx=0x7ff8bc008040).stop 2021-11-20T07:40:40.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 msgr2=0x7ff8c0106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:40.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7ff8b0009f50 tx=0x7ff8b0009b30).stop 2021-11-20T07:40:40.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 shutdown_connections 2021-11-20T07:40:40.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff8ac05c2a0 0x7ff8ac05e760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:40.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8c00f66a0 0x7ff8c0106ec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:40.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 --2- 172.21.15.71:0/3628122293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8c00f7940 0x7ff8c01073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:40.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 >> 172.21.15.71:0/3628122293 conn(0x7ff8c00f1da0 msgr2=0x7ff8c00fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:40.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 shutdown_connections 2021-11-20T07:40:40.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:40.396+0000 7ff8c790b700 1 -- 172.21.15.71:0/3628122293 wait complete. 2021-11-20T07:40:40.401 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:40.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:40.410 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:41.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:41.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/695257798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 msgr2=0x7f9fac0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:41.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/695257798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac0f59c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f9f9c004660 tx=0x7f9f9c009b30).stop 2021-11-20T07:40:41.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/695257798 shutdown_connections 2021-11-20T07:40:41.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/695257798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 0x7f9fac0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/695257798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac0f59c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/695257798 >> 172.21.15.71:0/695257798 conn(0x7f9fac0f0ce0 msgr2=0x7f9fac0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:41.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/695257798 shutdown_connections 2021-11-20T07:40:41.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/695257798 wait complete. 2021-11-20T07:40:41.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 Processor -- start 2021-11-20T07:40:41.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.536+0000 7f9fb0eaa700 1 -- start start 2021-11-20T07:40:41.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:41.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 0x7f9fac10bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:41.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac10c1d0 con 0x7f9fac0f55a0 2021-11-20T07:40:41.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac10c310 con 0x7f9fac0f6840 2021-11-20T07:40:41.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:41.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48494/0 (socket says 172.21.15.71:48494) 2021-11-20T07:40:41.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 -- 172.21.15.71:0/4017913282 learned_addr learned my addr 172.21.15.71:0/4017913282 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:41.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fa9d9b700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 0x7f9fac10bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:41.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 -- 172.21.15.71:0/4017913282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 msgr2=0x7f9fac10bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:41.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 0x7f9fac10bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 -- 172.21.15.71:0/4017913282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c005040 con 0x7f9fac0f55a0 2021-11-20T07:40:41.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9faa59c700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9f9c009f50 tx=0x7f9f9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:41.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f9c018070 con 0x7f9fac0f55a0 2021-11-20T07:40:41.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f9c013410 con 0x7f9fac0f55a0 2021-11-20T07:40:41.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f9c01c620 con 0x7f9fac0f55a0 2021-11-20T07:40:41.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac10c510 con 0x7f9fac0f55a0 2021-11-20T07:40:41.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.537+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac10ca40 con 0x7f9fac0f55a0 2021-11-20T07:40:41.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.538+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac027a00 con 0x7f9fac0f55a0 2021-11-20T07:40:41.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.539+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9f9c01d420 con 0x7f9fac0f55a0 2021-11-20T07:40:41.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.539+0000 7f9f9affd700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 0x7f9f9405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:41.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.539+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f9f9c078030 con 0x7f9fac0f55a0 2021-11-20T07:40:41.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.539+0000 7f9fa9d9b700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 0x7f9f9405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:41.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.539+0000 7f9fa9d9b700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 0x7f9f9405e760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9fa00041d0 tx=0x7f9fa0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:41.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.544+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f9c04aa40 con 0x7f9fac0f55a0 2021-11-20T07:40:41.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.728+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fac0fdd30 con 0x7f9fac0f55a0 2021-11-20T07:40:41.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.728+0000 7f9f9affd700 1 -- 172.21.15.71:0/4017913282 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f9f9c04e060 con 0x7f9fac0f55a0 2021-11-20T07:40:41.730 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:41.730 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 msgr2=0x7f9f9405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 0x7f9f9405e760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9fa00041d0 tx=0x7f9fa0009040).stop 2021-11-20T07:40:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 msgr2=0x7f9fac05ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9f9c009f50 tx=0x7f9f9c009b30).stop 2021-11-20T07:40:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 shutdown_connections 2021-11-20T07:40:41.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9f9405c2a0 0x7f9f9405e760 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fac0f55a0 0x7f9fac05ff00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 --2- 172.21.15.71:0/4017913282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fac0f6840 0x7f9fac10bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:41.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 >> 172.21.15.71:0/4017913282 conn(0x7f9fac0f0ce0 msgr2=0x7f9fac0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:41.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 shutdown_connections 2021-11-20T07:40:41.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:41.730+0000 7f9fb0eaa700 1 -- 172.21.15.71:0/4017913282 wait complete. 2021-11-20T07:40:41.736 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:41.741 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:41 smithi071 conmon[33687]: audit 2021-11-20T07:40:40.395765+0000 mon.smithi071 (mon.0) 668 : audit [DBG] from='client.? 172.21.15.71:0/3628122293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:41.742 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:41 smithi071 conmon[33687]: cluster 2021-11-20T07:40:40.396441+0000 mgr.smithi071.ynqmwn (mgr.14162) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 4.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T07:40:41.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:41.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:41 smithi112 conmon[36062]: audit 2021-11-20T07:40:40.395765+0000 mon.smithi071 (mon.0) 668 : audit [DBG] 2021-11-20T07:40:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:41 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/3628122293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:41 smithi112 conmon[36062]: cluster 2021-11-20T07:40:40.396441+0000 mgr.smithi071.ynqmwn (mgr.14162) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 4.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T07:40:42.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:42 smithi071 conmon[33687]: audit 2021-11-20T07:40:41.728851+0000 mon.smithi071 (mon.0) 669 : audit [DBG] from='client.? 172.21.15.71:0/4017913282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:42.744 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:42 smithi112 conmon[36062]: audit 2021-11-20T07:40:41.728851+0000 mon.smithi071 (mon.0) 669 : audit [DBG] from='client.? 172.21.15.71:0/4017913282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 -- 172.21.15.71:0/2395809233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 msgr2=0x7fc5380f7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:42.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 --2- 172.21.15.71:0/2395809233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5380f7d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc52c004660 tx=0x7fc52c009b30).stop 2021-11-20T07:40:42.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 -- 172.21.15.71:0/2395809233 shutdown_connections 2021-11-20T07:40:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 --2- 172.21.15.71:0/2395809233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5380f7d70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 --2- 172.21.15.71:0/2395809233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 0x7fc5380f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 -- 172.21.15.71:0/2395809233 >> 172.21.15.71:0/2395809233 conn(0x7fc5380f1db0 msgr2=0x7fc5380f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 -- 172.21.15.71:0/2395809233 shutdown_connections 2021-11-20T07:40:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.870+0000 7fc53dee3700 1 -- 172.21.15.71:0/2395809233 wait complete. 2021-11-20T07:40:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 Processor -- start 2021-11-20T07:40:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 -- start start 2021-11-20T07:40:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 0x7fc538106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:42.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5381079e0 con 0x7fc5380f7910 2021-11-20T07:40:42.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc53dee3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538107b20 con 0x7fc5380f6670 2021-11-20T07:40:42.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc536ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:42.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc536ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48508/0 (socket says 172.21.15.71:48508) 2021-11-20T07:40:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.871+0000 7fc536ffd700 1 -- 172.21.15.71:0/1900884880 learned_addr learned my addr 172.21.15.71:0/1900884880 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc5377fe700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 0x7fc538106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc536ffd700 1 -- 172.21.15.71:0/1900884880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 msgr2=0x7fc538106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc536ffd700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 0x7fc538106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc536ffd700 1 -- 172.21.15.71:0/1900884880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c005040 con 0x7fc5380f7910 2021-11-20T07:40:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc536ffd700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc52c009830 tx=0x7fc52c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52c018070 con 0x7fc5380f7910 2021-11-20T07:40:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53810c560 con 0x7fc5380f7910 2021-11-20T07:40:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc52c013410 con 0x7fc5380f7910 2021-11-20T07:40:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52c01c620 con 0x7fc5380f7910 2021-11-20T07:40:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.872+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53810cac0 con 0x7fc5380f7910 2021-11-20T07:40:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.873+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc538027a00 con 0x7fc5380f7910 2021-11-20T07:40:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.873+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc52c01c8e0 con 0x7fc5380f7910 2021-11-20T07:40:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.874+0000 7fc51ffff700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 0x7fc52005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.874+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc52c078030 con 0x7fc5380f7910 2021-11-20T07:40:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.874+0000 7fc5377fe700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 0x7fc52005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.874+0000 7fc5377fe700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 0x7fc52005e760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc5380f7770 tx=0x7fc528008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:42.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:42.877+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc52c04aa40 con 0x7fc5380f7910 2021-11-20T07:40:43.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.061+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5380feea0 con 0x7fc5380f7910 2021-11-20T07:40:43.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.061+0000 7fc51ffff700 1 -- 172.21.15.71:0/1900884880 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7fc52c04e060 con 0x7fc5380f7910 2021-11-20T07:40:43.063 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:43.063 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:43.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 msgr2=0x7fc52005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:43.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 0x7fc52005e760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc5380f7770 tx=0x7fc528008040).stop 2021-11-20T07:40:43.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 msgr2=0x7fc5381073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:43.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc52c009830 tx=0x7fc52c009b30).stop 2021-11-20T07:40:43.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 shutdown_connections 2021-11-20T07:40:43.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5380f6670 0x7fc538106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:43.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc52005c2a0 0x7fc52005e760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:43.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 --2- 172.21.15.71:0/1900884880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5380f7910 0x7fc5381073a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:43.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 >> 172.21.15.71:0/1900884880 conn(0x7fc5380f1db0 msgr2=0x7fc5380fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:43.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 shutdown_connections 2021-11-20T07:40:43.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:43.063+0000 7fc53dee3700 1 -- 172.21.15.71:0/1900884880 wait complete. 2021-11-20T07:40:43.067 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:43.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:43.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:43 smithi112 conmon[36062]: cluster 2021-11-20T07:40:42.396969+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:40:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:43 smithi112 conmon[36062]: 262 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:40:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:43 smithi112 conmon[36062]: audit 2021-11-20T07:40:43.061391+0000 mon.smithi071 (mon.0) 670 : audit [DBG] from='client.? 172.21.15.71:0/1900884880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:43.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:43 smithi071 conmon[33687]: cluster 2021-11-20T07:40:42.396969+0000 mgr.smithi071.ynqmwn (mgr.14162) 262 : 2021-11-20T07:40:43.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:43 smithi071 conmon[33687]: cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:40:43.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:43 smithi071 conmon[33687]: audit 2021-11-20T07:40:43.061391+0000 mon.smithi071 (mon.0) 670 : audit [DBG] from='client.? 172.21.15.71:0/1900884880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:44.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:44.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 -- 172.21.15.71:0/2228018733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f4840 msgr2=0x7f3d4c0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:44.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2228018733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f4c60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f3d3c004660 tx=0x7f3d3c009b30).stop 2021-11-20T07:40:44.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 -- 172.21.15.71:0/2228018733 shutdown_connections 2021-11-20T07:40:44.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2228018733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2228018733 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f4c60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.201+0000 7f3d517bc700 1 -- 172.21.15.71:0/2228018733 >> 172.21.15.71:0/2228018733 conn(0x7f3d4c0f0360 msgr2=0x7f3d4c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:44.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 -- 172.21.15.71:0/2228018733 shutdown_connections 2021-11-20T07:40:44.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 -- 172.21.15.71:0/2228018733 wait complete. 2021-11-20T07:40:44.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 Processor -- start 2021-11-20T07:40:44.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 -- start start 2021-11-20T07:40:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0f5dd0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.202+0000 7f3d517bc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0f5f10 con 0x7f3d4c0f4840 2021-11-20T07:40:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48520/0 (socket says 172.21.15.71:48520) 2021-11-20T07:40:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 -- 172.21.15.71:0/2204521250 learned_addr learned my addr 172.21.15.71:0/2204521250 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4bfff700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:44.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 -- 172.21.15.71:0/2204521250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 msgr2=0x7f3d4c0f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:44.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 -- 172.21.15.71:0/2204521250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c005040 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4bfff700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:40:44.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d4b7fe700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3d40002700 tx=0x7f3d40009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:44.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d40009320 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d4000f040 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c0f61f0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d400147c0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.203+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c10edc0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.211 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.204+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c045ad0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.204+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3d400077b0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.205+0000 7f3d48ff9700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 0x7f3d3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:44.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.205+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3d400767a0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.205+0000 7f3d4bfff700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 0x7f3d3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:44.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.205+0000 7f3d4bfff700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 0x7f3d3405e7c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f3d4c0f7180 tx=0x7f3d3c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:44.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.209+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d400491b0 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.392+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d4c0fee50 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.392+0000 7f3d48ff9700 1 -- 172.21.15.71:0/2204521250 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f3d4001b300 con 0x7f3d4c0f51e0 2021-11-20T07:40:44.394 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:44.394 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:44.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 msgr2=0x7f3d3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:44.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 0x7f3d3405e7c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f3d4c0f7180 tx=0x7f3d3c009b30).stop 2021-11-20T07:40:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 msgr2=0x7f3d4c0f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3d40002700 tx=0x7f3d40009d60).stop 2021-11-20T07:40:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 shutdown_connections 2021-11-20T07:40:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d4c0f4840 0x7f3d4c0f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d3405c300 0x7f3d3405e7c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 --2- 172.21.15.71:0/2204521250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d4c0f51e0 0x7f3d4c0f5870 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:44.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 >> 172.21.15.71:0/2204521250 conn(0x7f3d4c0f0360 msgr2=0x7f3d4c0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:44.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 shutdown_connections 2021-11-20T07:40:44.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:44.394+0000 7f3d517bc700 1 -- 172.21.15.71:0/2204521250 wait complete. 2021-11-20T07:40:44.398 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:44.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:44.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:44.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:44 smithi112 conmon[36062]: audit 2021-11-20T07:40:44.392710+0000 mon.smithi071 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.71:0/2204521250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:44.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:44 smithi071 conmon[33687]: audit 2021-11-20T07:40:44.392710+0000 mon.smithi071 (mon. 2021-11-20T07:40:44.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:44 smithi071 conmon[33687]: 0) 671 : audit [DBG] from='client.? 172.21.15.71:0/2204521250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:45.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:45.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 -- 172.21.15.71:0/2113902312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 msgr2=0x7f69c40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:45.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 --2- 172.21.15.71:0/2113902312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c40f4be0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f69b4004660 tx=0x7f69b4009b30).stop 2021-11-20T07:40:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 -- 172.21.15.71:0/2113902312 shutdown_connections 2021-11-20T07:40:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 --2- 172.21.15.71:0/2113902312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c40f4be0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 --2- 172.21.15.71:0/2113902312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c40f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 -- 172.21.15.71:0/2113902312 >> 172.21.15.71:0/2113902312 conn(0x7f69c40f0440 msgr2=0x7f69c40f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 -- 172.21.15.71:0/2113902312 shutdown_connections 2021-11-20T07:40:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.534+0000 7f69cb350700 1 -- 172.21.15.71:0/2113902312 wait complete. 2021-11-20T07:40:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 Processor -- start 2021-11-20T07:40:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 -- start start 2021-11-20T07:40:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c410def0 con 0x7f69c40f4760 2021-11-20T07:40:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69cb350700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c410e030 con 0x7f69c40f5a10 2021-11-20T07:40:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48528/0 (socket says 172.21.15.71:48528) 2021-11-20T07:40:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 -- 172.21.15.71:0/1279395690 learned_addr learned my addr 172.21.15.71:0/1279395690 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c88eb700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 -- 172.21.15.71:0/1279395690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 msgr2=0x7f69c410d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c410d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 -- 172.21.15.71:0/1279395690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4005040 con 0x7f69c40f4760 2021-11-20T07:40:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c88eb700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:40:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.535+0000 7f69c90ec700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f69c00026d0 tx=0x7f69c0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.536+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c0009320 con 0x7f69c40f4760 2021-11-20T07:40:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.536+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69c000f040 con 0x7f69c40f4760 2021-11-20T07:40:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.536+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c4112ad0 con 0x7f69c40f4760 2021-11-20T07:40:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.536+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c00147e0 con 0x7f69c40f4760 2021-11-20T07:40:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.536+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c4113010 con 0x7f69c40f4760 2021-11-20T07:40:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.537+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c4027a00 con 0x7f69c40f4760 2021-11-20T07:40:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.537+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f69c0009480 con 0x7f69c40f4760 2021-11-20T07:40:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.537+0000 7f69b9ffb700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 0x7f69b005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.538+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f69c00768b0 con 0x7f69c40f4760 2021-11-20T07:40:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.538+0000 7f69c88eb700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 0x7f69b005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:45.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.538+0000 7f69c88eb700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 0x7f69b005e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f69b4005010 tx=0x7f69b4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:45.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.541+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69c00492c0 con 0x7f69c40f4760 2021-11-20T07:40:45.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.726+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69c4105480 con 0x7f69c40f4760 2021-11-20T07:40:45.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.726+0000 7f69b9ffb700 1 -- 172.21.15.71:0/1279395690 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f69c004c8e0 con 0x7f69c40f4760 2021-11-20T07:40:45.727 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:45.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:45.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 msgr2=0x7f69b005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:45.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 0x7f69b005e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f69b4005010 tx=0x7f69b4009b30).stop 2021-11-20T07:40:45.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 msgr2=0x7f69c410d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:45.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f69c00026d0 tx=0x7f69c0009de0).stop 2021-11-20T07:40:45.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 shutdown_connections 2021-11-20T07:40:45.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f69b005c2a0 0x7f69b005e760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69c40f4760 0x7f69c410d450 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 --2- 172.21.15.71:0/1279395690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c40f5a10 0x7f69c410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:45.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 >> 172.21.15.71:0/1279395690 conn(0x7f69c40f0440 msgr2=0x7f69c40f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:45.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 shutdown_connections 2021-11-20T07:40:45.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:45.728+0000 7f69cb350700 1 -- 172.21.15.71:0/1279395690 wait complete. 2021-11-20T07:40:45.731 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:45.739 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:45 smithi071 conmon[33687]: cluster 2021-11-20T07:40:44.397436+0000 mgr.smithi071.ynqmwn (mgr.14162) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 369 B/s wr, 1 op/s 2021-11-20T07:40:45.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:45.741 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:45.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:45 smithi112 conmon[36062]: cluster 2021-11-20T07:40:44.397436+0000 mgr.smithi071.ynqmwn (mgr.14162) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 369 B/s wr, 1 op/s 2021-11-20T07:40:46.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:46 smithi071 conmon[33687]: audit 2021-11-20T07:40:45.726677+0000 mon.smithi071 (mon.0) 672 2021-11-20T07:40:46.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:46 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1279395690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:46.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:46.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:46 smithi112 conmon[36062]: audit 2021-11-20T07:40:45.726677+0000 mon.smithi071 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.71:0/1279395690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:46.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- 172.21.15.71:0/3800717896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f6820 msgr2=0x7f8f1c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:46.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 --2- 172.21.15.71:0/3800717896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f6c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8f0c004660 tx=0x7f8f0c009b30).stop 2021-11-20T07:40:46.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- 172.21.15.71:0/3800717896 shutdown_connections 2021-11-20T07:40:46.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 --2- 172.21.15.71:0/3800717896 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f6c80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:46.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 --2- 172.21.15.71:0/3800717896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:46.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- 172.21.15.71:0/3800717896 >> 172.21.15.71:0/3800717896 conn(0x7f8f1c0f0ce0 msgr2=0x7f8f1c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:46.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- 172.21.15.71:0/3800717896 shutdown_connections 2021-11-20T07:40:46.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- 172.21.15.71:0/3800717896 wait complete. 2021-11-20T07:40:46.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 Processor -- start 2021-11-20T07:40:46.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.862+0000 7f8f20e88700 1 -- start start 2021-11-20T07:40:46.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:46.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:46.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0fa2d0 con 0x7f8f1c0f5580 2021-11-20T07:40:46.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0f8870 con 0x7f8f1c0f6820 2021-11-20T07:40:46.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:46.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48540/0 (socket says 172.21.15.71:48540) 2021-11-20T07:40:46.868 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 -- 172.21.15.71:0/2389489655 learned_addr learned my addr 172.21.15.71:0/2389489655 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 -- 172.21.15.71:0/2389489655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f6820 msgr2=0x7f8f1c0f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f19d9b700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:46.869 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 -- 172.21.15.71:0/2389489655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c005040 con 0x7f8f1c0f5580 2021-11-20T07:40:46.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f1a59c700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f8f1000c790 tx=0x7f8f10007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:46.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1000e680 con 0x7f8f1c0f5580 2021-11-20T07:40:46.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f1000ec90 con 0x7f8f1c0f5580 2021-11-20T07:40:46.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c0f8af0 con 0x7f8f1c0f5580 2021-11-20T07:40:46.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f10012cb0 con 0x7f8f1c0f5580 2021-11-20T07:40:46.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.863+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c0f90b0 con 0x7f8f1c0f5580 2021-11-20T07:40:46.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.864+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c0fdd10 con 0x7f8f1c0f5580 2021-11-20T07:40:46.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.865+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f10010070 con 0x7f8f1c0f5580 2021-11-20T07:40:46.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.865+0000 7f8f0affd700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 0x7f8f0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:46.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.865+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f8f100761a0 con 0x7f8f1c0f5580 2021-11-20T07:40:46.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.865+0000 7f8f19d9b700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 0x7f8f0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:46.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.865+0000 7f8f19d9b700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 0x7f8f0405e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8f0c009d60 tx=0x7f8f0c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:46.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:46.869+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f10048bd0 con 0x7f8f1c0f5580 2021-11-20T07:40:47.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.050+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f1c10f3d0 con 0x7f8f1c0f5580 2021-11-20T07:40:47.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.051+0000 7f8f0affd700 1 -- 172.21.15.71:0/2389489655 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f8f1004c1f0 con 0x7f8f1c0f5580 2021-11-20T07:40:47.052 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:47.052 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:47.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.052+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 msgr2=0x7f8f0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:47.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.052+0000 7f8f20e88700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 0x7f8f0405e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8f0c009d60 tx=0x7f8f0c009b30).stop 2021-11-20T07:40:47.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.052+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 msgr2=0x7f8f1c0f9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:47.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.052+0000 7f8f20e88700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f8f1000c790 tx=0x7f8f10007d90).stop 2021-11-20T07:40:47.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 shutdown_connections 2021-11-20T07:40:47.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f0405c2a0 0x7f8f0405e760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:47.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f1c0f5580 0x7f8f1c0f9d20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:47.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 --2- 172.21.15.71:0/2389489655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f1c0f6820 0x7f8f1c0f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:47.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 >> 172.21.15.71:0/2389489655 conn(0x7f8f1c0f0ce0 msgr2=0x7f8f1c05faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:47.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 shutdown_connections 2021-11-20T07:40:47.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:47.053+0000 7f8f20e88700 1 -- 172.21.15.71:0/2389489655 wait complete. 2021-11-20T07:40:47.059 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:47.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:47.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:47 smithi112 conmon[36062]: cluster 2021-11-20T07:40:46.398033+0000 mgr.smithi071.ynqmwn (mgr.14162) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-20T07:40:47.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:47 smithi112 conmon[36062]: audit 2021-11-20T07:40:47.051198+0000 mon.smithi071 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.71:0/2389489655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:47 smithi071 conmon[33687]: cluster 2021-11-20T07:40:46.398033+0000 mgr.smithi071.ynqmwn (mgr.14162) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-20T07:40:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:47 smithi071 conmon[33687]: audit 2021-11- 2021-11-20T07:40:47.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:47 smithi071 conmon[33687]: 20T07:40:47.051198+0000 mon.smithi071 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.71:0/2389489655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:48.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:48.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- 172.21.15.71:0/1676357523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f55a0 msgr2=0x7fba380f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:48.191 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 --2- 172.21.15.71:0/1676357523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f55a0 0x7fba380f59c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fba34004660 tx=0x7fba34009b30).stop 2021-11-20T07:40:48.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- 172.21.15.71:0/1676357523 shutdown_connections 2021-11-20T07:40:48.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 --2- 172.21.15.71:0/1676357523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f6840 0x7fba380f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 --2- 172.21.15.71:0/1676357523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f55a0 0x7fba380f59c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.192 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- 172.21.15.71:0/1676357523 >> 172.21.15.71:0/1676357523 conn(0x7fba380f0ce0 msgr2=0x7fba380f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:48.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- 172.21.15.71:0/1676357523 shutdown_connections 2021-11-20T07:40:48.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- 172.21.15.71:0/1676357523 wait complete. 2021-11-20T07:40:48.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 Processor -- start 2021-11-20T07:40:48.193 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 -- start start 2021-11-20T07:40:48.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 0x7fba3805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:48.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.188+0000 7fba4078b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:48.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba4078b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38107aa0 con 0x7fba380f6840 2021-11-20T07:40:48.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba4078b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3810e5b0 con 0x7fba380f55a0 2021-11-20T07:40:48.194 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:48.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48550/0 (socket says 172.21.15.71:48550) 2021-11-20T07:40:48.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 -- 172.21.15.71:0/1603850075 learned_addr learned my addr 172.21.15.71:0/1603850075 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:48.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3e527700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 0x7fba3805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:48.195 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 -- 172.21.15.71:0/1603850075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 msgr2=0x7fba3805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:48.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 0x7fba3805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 -- 172.21.15.71:0/1603850075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34005040 con 0x7fba380f6840 2021-11-20T07:40:48.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3e527700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 0x7fba3805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:40:48.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba3dd26700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fba28002700 tx=0x7fba28009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:48.196 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba28009320 con 0x7fba380f6840 2021-11-20T07:40:48.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3810e750 con 0x7fba380f6840 2021-11-20T07:40:48.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba2800f040 con 0x7fba380f6840 2021-11-20T07:40:48.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba280148a0 con 0x7fba380f6840 2021-11-20T07:40:48.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.189+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3810ed10 con 0x7fba380f6840 2021-11-20T07:40:48.197 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.190+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38027a00 con 0x7fba380f6840 2021-11-20T07:40:48.198 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.191+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fba280077b0 con 0x7fba380f6840 2021-11-20T07:40:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.191+0000 7fba2effd700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 0x7fba2405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.191+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fba28076760 con 0x7fba380f6840 2021-11-20T07:40:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.191+0000 7fba3e527700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 0x7fba2405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.191+0000 7fba3e527700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 0x7fba2405e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fba34004e30 tx=0x7fba34009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.195+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba28049170 con 0x7fba380f6840 2021-11-20T07:40:48.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.394+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba380fdd30 con 0x7fba380f6840 2021-11-20T07:40:48.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.394+0000 7fba2effd700 1 -- 172.21.15.71:0/1603850075 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2958 (secure 0 0 0) 0x7fba2804c790 con 0x7fba380f6840 2021-11-20T07:40:48.396 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:48.396 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":33},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:37.422479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14566,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14566":{"gid":14566,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":41,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.71:6829/1481695887","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1481695887},{"type":"v1","addr":"172.21.15.71:6829","nonce":1481695887}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:48.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 msgr2=0x7fba2405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:48.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 0x7fba2405e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fba34004e30 tx=0x7fba34009b30).stop 2021-11-20T07:40:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 msgr2=0x7fba3810e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fba28002700 tx=0x7fba28009de0).stop 2021-11-20T07:40:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 shutdown_connections 2021-11-20T07:40:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba380f55a0 0x7fba3805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fba2405c2a0 0x7fba2405e760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 --2- 172.21.15.71:0/1603850075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba380f6840 0x7fba3810e0c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:48.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 >> 172.21.15.71:0/1603850075 conn(0x7fba380f0ce0 msgr2=0x7fba380fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:48.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 shutdown_connections 2021-11-20T07:40:48.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:48.396+0000 7fba4078b700 1 -- 172.21.15.71:0/1603850075 wait complete. 2021-11-20T07:40:48.400 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2021-11-20T07:40:48.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:37.422479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14566, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14566': {'gid': 14566, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 41, 'state': 'up:clientreplay', 'state_seq': 11, 'addr': '172.21.15.71:6829/1481695887', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1481695887}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1481695887}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:48.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14566:up:clientreplay 2021-11-20T07:40:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: cluster 2021-11-20T07:40:48.332544+0000 mon.smithi071 (mon.0) 674 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:40:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: cluster 2021-11-20T07:40:48.336907+0000 mon.smithi071 (mon.0) 675 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T07:40:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: audit 2021-11-20T07:40:48.394604+0000 mon.smithi071 (mon.0) 676 2021-11-20T07:40:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/1603850075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:40:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: -11-20T07:40:48.398373+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:40:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:48 smithi112 conmon[36062]: ) 265 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 307 B/s wr, 0 op/s 2021-11-20T07:40:48.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: cluster 2021-11-20T07:40:48.332544+0000 mon.smithi071 ( 2021-11-20T07:40:48.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: mon.0) 674 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:40:48.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: cluster 2021-11-20T07:40:48.336907+0000 mon.smithi071 (mon.0) 675 : cluster 2021-11-20T07:40:48.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T07:40:48.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:40:48.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: T07:40:48.394604+0000 mon.smithi071 (mon.0) 676 : 2021-11-20T07:40:48.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: audit [DBG] from='client.? 172.21.15.71:0/1603850075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:48.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: cluster 2021-11-20T07:40 2021-11-20T07:40:48.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: :48.398373+0000 mgr.smithi071.ynqmwn (mgr.14162) 265 2021-11-20T07:40:48.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:48 smithi071 conmon[33687]: : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 4.9 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 307 B/s wr, 0 op/s 2021-11-20T07:40:49.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:49.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 -- 172.21.15.71:0/896634460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 msgr2=0x7f17e40f7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:49.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 --2- 172.21.15.71:0/896634460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e40f7d70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f17d4004660 tx=0x7f17d4009b30).stop 2021-11-20T07:40:49.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 -- 172.21.15.71:0/896634460 shutdown_connections 2021-11-20T07:40:49.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 --2- 172.21.15.71:0/896634460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e40f7d70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 --2- 172.21.15.71:0/896634460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 0x7f17e40f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.538+0000 7f17ec188700 1 -- 172.21.15.71:0/896634460 >> 172.21.15.71:0/896634460 conn(0x7f17e40f1db0 msgr2=0x7f17e40f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:49.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 -- 172.21.15.71:0/896634460 shutdown_connections 2021-11-20T07:40:49.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 -- 172.21.15.71:0/896634460 wait complete. 2021-11-20T07:40:49.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 Processor -- start 2021-11-20T07:40:49.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 -- start start 2021-11-20T07:40:49.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 0x7f17e4106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:49.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:49.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e4107920 con 0x7f17e40f7910 2021-11-20T07:40:49.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17ec188700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e4107a60 con 0x7f17e40f6670 2021-11-20T07:40:49.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17e9723700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:49.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17e9723700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48570/0 (socket says 172.21.15.71:48570) 2021-11-20T07:40:49.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17e9723700 1 -- 172.21.15.71:0/1038128841 learned_addr learned my addr 172.21.15.71:0/1038128841 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:49.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.539+0000 7f17e9f24700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 0x7f17e4106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:49.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17e9723700 1 -- 172.21.15.71:0/1038128841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 msgr2=0x7f17e4106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:49.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17e9723700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 0x7f17e4106e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17e9723700 1 -- 172.21.15.71:0/1038128841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d4005040 con 0x7f17e40f7910 2021-11-20T07:40:49.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17e9723700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f17d4009f50 tx=0x7f17d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:49.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d4018070 con 0x7f17e40f7910 2021-11-20T07:40:49.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17d4013410 con 0x7f17e40f7910 2021-11-20T07:40:49.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d401c3c0 con 0x7f17e40f7910 2021-11-20T07:40:49.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e410c4a0 con 0x7f17e40f7910 2021-11-20T07:40:49.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.540+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e410ca60 con 0x7f17e40f7910 2021-11-20T07:40:49.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.541+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e4027a00 con 0x7f17e40f7910 2021-11-20T07:40:49.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.541+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f17d401d420 con 0x7f17e40f7910 2021-11-20T07:40:49.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.542+0000 7f17da7fc700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 0x7f17d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:49.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.542+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f17d4077d90 con 0x7f17e40f7910 2021-11-20T07:40:49.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.542+0000 7f17e9f24700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 0x7f17d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:49.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.542+0000 7f17e9f24700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 0x7f17d005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f17e40f7770 tx=0x7f17e000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:49.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.546+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17d404a7d0 con 0x7f17e40f7910 2021-11-20T07:40:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.732+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e40fee00 con 0x7f17e40f7910 2021-11-20T07:40:49.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.732+0000 7f17da7fc700 1 -- 172.21.15.71:0/1038128841 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f17d404ddf0 con 0x7f17e40f7910 2021-11-20T07:40:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:49.734 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":50},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:49.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 msgr2=0x7f17d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:49.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 0x7f17d005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f17e40f7770 tx=0x7f17e000b040).stop 2021-11-20T07:40:49.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 msgr2=0x7f17e4107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:49.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f17d4009f50 tx=0x7f17d4009b30).stop 2021-11-20T07:40:49.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 shutdown_connections 2021-11-20T07:40:49.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17e40f6670 0x7f17e4106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f17d005c2a0 0x7f17d005e760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 --2- 172.21.15.71:0/1038128841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17e40f7910 0x7f17e4107370 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:49.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 >> 172.21.15.71:0/1038128841 conn(0x7f17e40f1db0 msgr2=0x7f17e40fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:49.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 shutdown_connections 2021-11-20T07:40:49.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:49.734+0000 7f17ec188700 1 -- 172.21.15.71:0/1038128841 wait complete. 2021-11-20T07:40:49.901 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 52 2021-11-20T07:40:49.902 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: cluster 2021-11-20T07:40:48.422441+0000 mon.smithi071 (mon.0) 677 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:boot 2021-11-20T07:40:49.902 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:40:49.902 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: 20T07:40:48.422506+0000 mon.smithi071 (mon.0) 678 : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:40:49.902 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: cluster 2021-11-20T07:40:48.422750+0000 mon.smithi071 (mon.0) 679 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.xtyomb=up:active} 3 up:standby, 1 failed 2021-11-20T07:40:49.903 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:40:49.903 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: 11-20T07:40:48.422833+0000 mon.smithi071 (mon.0) 680 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:40:49.903 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:40:49.903 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: -11-20T07 2021-11-20T07:40:49.904 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: :40:48.428037+0000 mon.smithi071 (mon.0) 681 : cluster [DBG] 2021-11-20T07:40:49.904 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:49 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:49.904 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: cluster 2021-11-20T07:40:48.422441+0000 mon.smithi071 ( 2021-11-20T07:40:49.904 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: mon.0) 677 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:boot 2021-11-20T07:40:49.905 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: cluster 2021-11-20T07:40:48.422506+0000 mon.smithi071 (mon.0) 678 2021-11-20T07:40:49.905 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:40:49.905 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:40:49.905 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: 40:48.422750+0000 mon.smithi071 (mon.0) 679 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.xtyomb=up:active} 3 up:standby, 1 failed 2021-11-20T07:40:49.905 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: audit 2021 2021-11-20T07:40:49.906 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: -11-20T07:40:48.422833+0000 2021-11-20T07:40:49.906 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: mon.smithi071 (mon.0) 680 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:40:49.906 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: cluster 2021-11-20T07:40:49.906 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: 2021-11-20T07:40 2021-11-20T07:40:49.907 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: :48.428037+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:49.907 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:49 smithi071 conmon[33687]: 681 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:49.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:49.908 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:50.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: cluster 2021-11-20T07:40:49.430832+0000 mon.smithi071 (mon.0) 682 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:resolve 2021-11-20T07:40:50.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: cluster 2021-11-20T07:40:49.430904+0000 mon.smithi071 (mon.0) 683 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:50.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: audit 2021-11-20T07:40:49. 2021-11-20T07:40:50.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: 732469+0000 mon.smithi071 (mon.0) 684 : audit [DBG] from='client.? 172.21.15.71:0/1038128841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:50.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: cluster 2021-11-20T07:40:50.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: 2021-11-20T07:40:50 2021-11-20T07:40:50.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: .398724+0000 mgr.smithi071.ynqmwn (mgr.14162) 266 2021-11-20T07:40:50.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:50 smithi112 conmon[36062]: : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T07:40:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: cluster 2021-11-20T07:40:49.430832+0000 mon.smithi071 (mon.0) 682 2021-11-20T07:40:50.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:resolve 2021-11-20T07:40:50.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: cluster 2021-11-20T07:40:49.430904+0000 mon.smithi071 (mon.0) 683 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:50.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: audit 2021-11-20T07:40:50.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: 2021-11-20T07:40:49.732469+0000 mon.smithi071 (mon.0) 684 : audit [DBG] from='client.? 172.21.15.71:0/1038128841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:50.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:40:50.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: T07:40:50.398724+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:40:50.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:50 smithi071 conmon[33687]: 266 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T07:40:50.909 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.036+0000 7fa08359e700 1 -- 172.21.15.71:0/4151980455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 msgr2=0x7fa0840f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.036+0000 7fa08359e700 1 --2- 172.21.15.71:0/4151980455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa0840f6e90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa074004660 tx=0x7fa074009b30).stop 2021-11-20T07:40:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.036+0000 7fa08359e700 1 -- 172.21.15.71:0/4151980455 shutdown_connections 2021-11-20T07:40:51.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 --2- 172.21.15.71:0/4151980455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 0x7fa0840f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 --2- 172.21.15.71:0/4151980455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa0840f6e90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.042 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- 172.21.15.71:0/4151980455 >> 172.21.15.71:0/4151980455 conn(0x7fa0840f0360 msgr2=0x7fa0840f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- 172.21.15.71:0/4151980455 shutdown_connections 2021-11-20T07:40:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- 172.21.15.71:0/4151980455 wait complete. 2021-11-20T07:40:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 Processor -- start 2021-11-20T07:40:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- start start 2021-11-20T07:40:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 0x7fa08410bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840f5310 con 0x7fa0840f6a70 2021-11-20T07:40:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa08359e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840f5450 con 0x7fa0840f5770 2021-11-20T07:40:51.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa081d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:51.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa081d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48582/0 (socket says 172.21.15.71:48582) 2021-11-20T07:40:51.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.037+0000 7fa081d9b700 1 -- 172.21.15.71:0/340415678 learned_addr learned my addr 172.21.15.71:0/340415678 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:51.045 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa08259c700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 0x7fa08410bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:51.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa081d9b700 1 -- 172.21.15.71:0/340415678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 msgr2=0x7fa08410bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:51.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa081d9b700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 0x7fa08410bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa081d9b700 1 -- 172.21.15.71:0/340415678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa074005040 con 0x7fa0840f6a70 2021-11-20T07:40:51.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa081d9b700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa078002700 tx=0x7fa078009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:51.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa078009320 con 0x7fa0840f6a70 2021-11-20T07:40:51.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08410e7d0 con 0x7fa0840f6a70 2021-11-20T07:40:51.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa07800f040 con 0x7fa0840f6a70 2021-11-20T07:40:51.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0780148e0 con 0x7fa0840f6a70 2021-11-20T07:40:51.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.038+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08410ec50 con 0x7fa0840f6a70 2021-11-20T07:40:51.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.039+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa084045ad0 con 0x7fa0840f6a70 2021-11-20T07:40:51.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.039+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa078009480 con 0x7fa0840f6a70 2021-11-20T07:40:51.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.040+0000 7fa072ffd700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 0x7fa06c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:51.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.040+0000 7fa08259c700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 0x7fa06c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:51.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.040+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa078076440 con 0x7fa0840f6a70 2021-11-20T07:40:51.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.040+0000 7fa08259c700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 0x7fa06c05e7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa0840f50a0 tx=0x7fa074009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:51.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.044+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa078048e00 con 0x7fa0840f6a70 2021-11-20T07:40:51.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.231+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa084027a00 con 0x7fa0840f6a70 2021-11-20T07:40:51.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.233+0000 7fa072ffd700 1 -- 172.21.15.71:0/340415678 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fa07804c420 con 0x7fa0840f6a70 2021-11-20T07:40:51.234 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:51.234 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":50},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:51.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.235+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 msgr2=0x7fa06c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:51.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.235+0000 7fa08359e700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 0x7fa06c05e7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa0840f50a0 tx=0x7fa074009b30).stop 2021-11-20T07:40:51.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.235+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 msgr2=0x7fa08410e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:51.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.235+0000 7fa08359e700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa078002700 tx=0x7fa078009e00).stop 2021-11-20T07:40:51.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 shutdown_connections 2021-11-20T07:40:51.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0840f5770 0x7fa08410bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa06c05c300 0x7fa06c05e7c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 --2- 172.21.15.71:0/340415678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0840f6a70 0x7fa08410e2e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:51.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 >> 172.21.15.71:0/340415678 conn(0x7fa0840f0360 msgr2=0x7fa0840fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:51.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 shutdown_connections 2021-11-20T07:40:51.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:51.236+0000 7fa08359e700 1 -- 172.21.15.71:0/340415678 wait complete. 2021-11-20T07:40:51.239 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 52 2021-11-20T07:40:51.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:51.250 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:51.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:51 smithi071 conmon[33687]: audit 2021-11-20T07:40:51.232519+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:51.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:51 smithi071 conmon[33687]: 685 : audit [DBG] from='client.? 172.21.15.71:0/340415678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:51.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:51 smithi112 conmon[36062]: audit 2021-11-20T07:40:51.232519+0000 mon.smithi071 (mon.0) 685 : audit [DBG] from='client.? 172.21.15.71:0/340415678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:52.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:52.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 -- 172.21.15.71:0/578029636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 msgr2=0x7f95f40f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:52.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/578029636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f40f5880 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f95e8004660 tx=0x7f95e8009b30).stop 2021-11-20T07:40:52.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 -- 172.21.15.71:0/578029636 shutdown_connections 2021-11-20T07:40:52.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/578029636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95f40f6700 0x7f95f40f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:52.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/578029636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f40f5880 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:52.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 -- 172.21.15.71:0/578029636 >> 172.21.15.71:0/578029636 conn(0x7f95f40f0ba0 msgr2=0x7f95f40f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:52.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 -- 172.21.15.71:0/578029636 shutdown_connections 2021-11-20T07:40:52.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.377+0000 7f95f9ee2700 1 -- 172.21.15.71:0/578029636 wait complete. 2021-11-20T07:40:52.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 Processor -- start 2021-11-20T07:40:52.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 -- start start 2021-11-20T07:40:52.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:52.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95f40f6700 0x7f95f410bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:52.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4060210 con 0x7f95f40f5460 2021-11-20T07:40:52.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f9ee2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4060350 con 0x7f95f40f6700 2021-11-20T07:40:52.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f8ee0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:52.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f8ee0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48590/0 (socket says 172.21.15.71:48590) 2021-11-20T07:40:52.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.378+0000 7f95f8ee0700 1 -- 172.21.15.71:0/3371616263 learned_addr learned my addr 172.21.15.71:0/3371616263 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:52.385 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f8ee0700 1 -- 172.21.15.71:0/3371616263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95f40f6700 msgr2=0x7f95f410bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:52.385 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f8ee0700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95f40f6700 0x7f95f410bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:52.385 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f8ee0700 1 -- 172.21.15.71:0/3371616263 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e8005040 con 0x7f95f40f5460 2021-11-20T07:40:52.385 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f8ee0700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f95e8000c00 tx=0x7f95e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e8019070 con 0x7f95f40f5460 2021-11-20T07:40:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95e8007e70 con 0x7f95f40f5460 2021-11-20T07:40:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f410c430 con 0x7f95f40f5460 2021-11-20T07:40:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e8014db0 con 0x7f95f40f5460 2021-11-20T07:40:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.379+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f410c990 con 0x7f95f40f5460 2021-11-20T07:40:52.387 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.380+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f4045ad0 con 0x7f95f40f5460 2021-11-20T07:40:52.388 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.380+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f95e8025070 con 0x7f95f40f5460 2021-11-20T07:40:52.388 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.380+0000 7f95f17fa700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 0x7f95e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:52.388 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.381+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f95e8077100 con 0x7f95f40f5460 2021-11-20T07:40:52.389 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.381+0000 7f95f3fff700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 0x7f95e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:52.389 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.381+0000 7f95f3fff700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 0x7f95e405e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f95e0004b70 tx=0x7f95e000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:52.389 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.384+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95e8049ac0 con 0x7f95f40f5460 2021-11-20T07:40:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:52 smithi071 conmon[33687]: cluster 2021-11-20T07:40:52.399134+0000 2021-11-20T07:40:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:52 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 267 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2021-11-20T07:40:52.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.576+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95f4027a00 con 0x7f95f40f5460 2021-11-20T07:40:52.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.576+0000 7f95f17fa700 1 -- 172.21.15.71:0/3371616263 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f95e804d0e0 con 0x7f95f40f5460 2021-11-20T07:40:52.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:52.577 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:52.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 msgr2=0x7f95e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:52.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 0x7f95e405e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f95e0004b70 tx=0x7f95e000b040).stop 2021-11-20T07:40:52.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 msgr2=0x7f95f41099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:52.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f95e8000c00 tx=0x7f95e800b040).stop 2021-11-20T07:40:52.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 shutdown_connections 2021-11-20T07:40:52.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f95e405c2a0 0x7f95e405e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:52.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95f40f5460 0x7f95f41099e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:52.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 --2- 172.21.15.71:0/3371616263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95f40f6700 0x7f95f410bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:52.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 >> 172.21.15.71:0/3371616263 conn(0x7f95f40f0ba0 msgr2=0x7f95f40fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:52.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 shutdown_connections 2021-11-20T07:40:52.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:52.578+0000 7f95f9ee2700 1 -- 172.21.15.71:0/3371616263 wait complete. 2021-11-20T07:40:52.582 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:52.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:52.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:52 smithi112 conmon[36062]: cluster 2021-11-20T07:40:52.399134+0000 mgr.smithi071.ynqmwn (mgr.14162) 267 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2021-11-20T07:40:53.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:53 smithi071 conmon[33687]: cluster 2021-11-20T07:40:52.437233+0000 mon.smithi071 (mon.0) 686 2021-11-20T07:40:53.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:53 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:standby 2021-11-20T07:40:53.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:53 smithi071 conmon[33687]: cluster 2021-11-20T07:40:52.437312+0000 mon.smithi071 (mon.0) 687 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:53.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:53 smithi071 conmon[33687]: audit 2021-11-20T07:40:52.576433+0000 mon.smithi071 (mon.0) 688 : audit [DBG] 2021-11-20T07:40:53.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:53 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/3371616263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:53.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:53.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.718+0000 7f9e1516a700 1 -- 172.21.15.71:0/3456383122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 msgr2=0x7f9e100f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:53.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.718+0000 7f9e1516a700 1 --2- 172.21.15.71:0/3456383122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e100f5910 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f9e00004660 tx=0x7f9e00009b30).stop 2021-11-20T07:40:53.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.718+0000 7f9e1516a700 1 -- 172.21.15.71:0/3456383122 shutdown_connections 2021-11-20T07:40:53.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.718+0000 7f9e1516a700 1 --2- 172.21.15.71:0/3456383122 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e100f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 --2- 172.21.15.71:0/3456383122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e100f5910 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- 172.21.15.71:0/3456383122 >> 172.21.15.71:0/3456383122 conn(0x7f9e100f0c30 msgr2=0x7f9e100f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:53.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- 172.21.15.71:0/3456383122 shutdown_connections 2021-11-20T07:40:53.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- 172.21.15.71:0/3456383122 wait complete. 2021-11-20T07:40:53.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 Processor -- start 2021-11-20T07:40:53.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- start start 2021-11-20T07:40:53.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:53.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e1010c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:53.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10060580 con 0x7f9e100f54f0 2021-11-20T07:40:53.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e1516a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100606c0 con 0x7f9e100f6790 2021-11-20T07:40:53.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.719+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:53.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48602/0 (socket says 172.21.15.71:48602) 2021-11-20T07:40:53.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 -- 172.21.15.71:0/2468285991 learned_addr learned my addr 172.21.15.71:0/2468285991 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:53.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0f7fe700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e1010c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:53.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 -- 172.21.15.71:0/2468285991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 msgr2=0x7f9e1010c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:53.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e1010c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 -- 172.21.15.71:0/2468285991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00005040 con 0x7f9e100f54f0 2021-11-20T07:40:53.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0f7fe700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e1010c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:40:53.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0ffff700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9e00009f50 tx=0x7f9e00009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:53.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e00018070 con 0x7f9e100f54f0 2021-11-20T07:40:53.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1010c7d0 con 0x7f9e100f54f0 2021-11-20T07:40:53.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e00013410 con 0x7f9e100f54f0 2021-11-20T07:40:53.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0001c620 con 0x7f9e100f54f0 2021-11-20T07:40:53.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.720+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1010cd30 con 0x7f9e100f54f0 2021-11-20T07:40:53.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.721+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10045ad0 con 0x7f9e100f54f0 2021-11-20T07:40:53.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.721+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e0001d420 con 0x7f9e100f54f0 2021-11-20T07:40:53.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.722+0000 7f9e0cff9700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 0x7f9df805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:53.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.722+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9e00078080 con 0x7f9e100f54f0 2021-11-20T07:40:53.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.722+0000 7f9e0f7fe700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 0x7f9df805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:53.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.722+0000 7f9e0f7fe700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 0x7f9df805e7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9e040041d0 tx=0x7f9e04009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:53.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.726+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e0004aa40 con 0x7f9e100f54f0 2021-11-20T07:40:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:53 smithi112 conmon[36062]: cluster 2021-11-20T07:40:52.437233+0000 mon.smithi071 (mon.0) 686 : cluster 2021-11-20T07:40:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:53 smithi112 conmon[36062]: [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:standby 2021-11-20T07:40:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:53 smithi112 conmon[36062]: cluster 2021-11-20T07:40:52.437312+0000 mon.smithi071 (mon.0) 687 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:40:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:53 smithi112 conmon[36062]: audit 2021-11-20T07:40:52.576433+0000 mon.smithi071 (mon.0) 688 : audit 2021-11-20T07:40:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:53 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/3371616263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:53.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.911+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e10027a00 con 0x7f9e100f54f0 2021-11-20T07:40:53.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.912+0000 7f9e0cff9700 1 -- 172.21.15.71:0/2468285991 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f9e0004e060 con 0x7f9e100f54f0 2021-11-20T07:40:53.915 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:53.915 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:53.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 msgr2=0x7f9df805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:53.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 0x7f9df805e7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9e040041d0 tx=0x7f9e04009040).stop 2021-11-20T07:40:53.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 msgr2=0x7f9e10109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:53.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9e00009f50 tx=0x7f9e00009b30).stop 2021-11-20T07:40:53.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 shutdown_connections 2021-11-20T07:40:53.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9df805c300 0x7f9df805e7c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100f54f0 0x7f9e10109d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 --2- 172.21.15.71:0/2468285991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e100f6790 0x7f9e1010c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:53.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 >> 172.21.15.71:0/2468285991 conn(0x7f9e100f0c30 msgr2=0x7f9e100fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:53.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 shutdown_connections 2021-11-20T07:40:53.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:53.913+0000 7f9e1516a700 1 -- 172.21.15.71:0/2468285991 wait complete. 2021-11-20T07:40:53.923 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:53.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:53.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:54.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:54 smithi112 conmon[36062]: audit 2021-11-20T07:40:53.912922+0000 mon.smithi071 (mon.0) 689 : audit [DBG] from='client.? 172.21.15.71:0/2468285991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:54.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:54 smithi112 conmon[36062]: cluster 2021-11-20T07:40:54.399544+0000 mgr.smithi071.ynqmwn (mgr.14162) 268 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:54 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:40:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:54 smithi071 conmon[33687]: T07:40:53.912922+0000 mon.smithi071 (mon.0) 689 : audit [DBG] from='client.? 172.21.15.71:0/2468285991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:54 smithi071 conmon[33687]: cluster 2021-11-20 2021-11-20T07:40:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:54 smithi071 conmon[33687]: T07:40:54.399544+0000 mgr.smithi071.ynqmwn (mgr.14162) 268 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:54.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:55.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.051+0000 7f159a3c4700 1 -- 172.21.15.71:0/3573901441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 msgr2=0x7f15940f6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:55.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.051+0000 7f159a3c4700 1 --2- 172.21.15.71:0/3573901441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f15940f6960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f1588004660 tx=0x7f1588009b30).stop 2021-11-20T07:40:55.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.051+0000 7f159a3c4700 1 -- 172.21.15.71:0/3573901441 shutdown_connections 2021-11-20T07:40:55.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 --2- 172.21.15.71:0/3573901441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f15940f6960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 --2- 172.21.15.71:0/3573901441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 0x7f15940f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 -- 172.21.15.71:0/3573901441 >> 172.21.15.71:0/3573901441 conn(0x7f15940f0410 msgr2=0x7f15940f2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:55.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 -- 172.21.15.71:0/3573901441 shutdown_connections 2021-11-20T07:40:55.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 -- 172.21.15.71:0/3573901441 wait complete. 2021-11-20T07:40:55.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 Processor -- start 2021-11-20T07:40:55.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 -- start start 2021-11-20T07:40:55.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 0x7f1594105d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:55.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.052+0000 7f159a3c4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:55.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f159a3c4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594106880 con 0x7f15940f6500 2021-11-20T07:40:55.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f159a3c4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15941069c0 con 0x7f15940f5b60 2021-11-20T07:40:55.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:55.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48614/0 (socket says 172.21.15.71:48614) 2021-11-20T07:40:55.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 -- 172.21.15.71:0/4253445469 learned_addr learned my addr 172.21.15.71:0/4253445469 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:55.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f1593fff700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 0x7f1594105d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:55.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 -- 172.21.15.71:0/4253445469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 msgr2=0x7f1594105d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:55.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 0x7f1594105d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 -- 172.21.15.71:0/4253445469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588005040 con 0x7f15940f6500 2021-11-20T07:40:55.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f15937fe700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1588009f50 tx=0x7f1588009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:55.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1588018070 con 0x7f15940f6500 2021-11-20T07:40:55.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159410b370 con 0x7f15940f6500 2021-11-20T07:40:55.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1588013410 con 0x7f15940f6500 2021-11-20T07:40:55.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f158801c3c0 con 0x7f15940f6500 2021-11-20T07:40:55.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.053+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159410b8d0 con 0x7f15940f6500 2021-11-20T07:40:55.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.054+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1594027a00 con 0x7f15940f6500 2021-11-20T07:40:55.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.055+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f158801d420 con 0x7f15940f6500 2021-11-20T07:40:55.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.055+0000 7f1590ff9700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 0x7f157c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:55.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.055+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1588077e40 con 0x7f15940f6500 2021-11-20T07:40:55.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.055+0000 7f1593fff700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 0x7f157c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:55.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.055+0000 7f1593fff700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 0x7f157c05e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1594107280 tx=0x7f158400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:55.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.059+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f158804a800 con 0x7f15940f6500 2021-11-20T07:40:55.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.243+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15940fdd20 con 0x7f15940f6500 2021-11-20T07:40:55.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.243+0000 7f1590ff9700 1 -- 172.21.15.71:0/4253445469 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f158804de20 con 0x7f15940f6500 2021-11-20T07:40:55.245 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:55.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:55.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.244+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 msgr2=0x7f157c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:55.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 0x7f157c05e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1594107280 tx=0x7f158400b040).stop 2021-11-20T07:40:55.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 msgr2=0x7f1594106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:55.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1588009f50 tx=0x7f1588009b30).stop 2021-11-20T07:40:55.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 shutdown_connections 2021-11-20T07:40:55.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15940f5b60 0x7f1594105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f157c05c2a0 0x7f157c05e760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 --2- 172.21.15.71:0/4253445469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15940f6500 0x7f1594106280 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:55.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 >> 172.21.15.71:0/4253445469 conn(0x7f15940f0410 msgr2=0x7f15940fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:55.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 shutdown_connections 2021-11-20T07:40:55.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:55.245+0000 7f159a3c4700 1 -- 172.21.15.71:0/4253445469 wait complete. 2021-11-20T07:40:55.250 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:55.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:55.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:55.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:55 smithi071 conmon[33687]: audit 2021-11-20T07:40:55.244443+0000 2021-11-20T07:40:55.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:55 smithi071 conmon[33687]: mon.smithi071 (mon.0) 690 : audit [DBG] from='client.? 172.21.15.71:0/4253445469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:55 smithi112 conmon[36062]: audit 2021-11-20T07:40:55.244443+0000 mon.smithi071 (mon.0) 690 : audit [DBG] from='client.? 172.21.15.71:0/4253445469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:56.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:56.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.384+0000 7fcf26679700 1 -- 172.21.15.71:0/1383798314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 msgr2=0x7fcf200f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:56.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.384+0000 7fcf26679700 1 --2- 172.21.15.71:0/1383798314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f6c90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fcf14004660 tx=0x7fcf14009b30).stop 2021-11-20T07:40:56.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- 172.21.15.71:0/1383798314 shutdown_connections 2021-11-20T07:40:56.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 --2- 172.21.15.71:0/1383798314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f6c90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 --2- 172.21.15.71:0/1383798314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 0x7fcf200f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- 172.21.15.71:0/1383798314 >> 172.21.15.71:0/1383798314 conn(0x7fcf200f0cd0 msgr2=0x7fcf200f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:56.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- 172.21.15.71:0/1383798314 shutdown_connections 2021-11-20T07:40:56.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- 172.21.15.71:0/1383798314 wait complete. 2021-11-20T07:40:56.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 Processor -- start 2021-11-20T07:40:56.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- start start 2021-11-20T07:40:56.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 0x7fcf200f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:56.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:56.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf200fa2e0 con 0x7fcf200f6830 2021-11-20T07:40:56.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.385+0000 7fcf26679700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf200f8880 con 0x7fcf200f5590 2021-11-20T07:40:56.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:56.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48624/0 (socket says 172.21.15.71:48624) 2021-11-20T07:40:56.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 -- 172.21.15.71:0/1317860269 learned_addr learned my addr 172.21.15.71:0/1317860269 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:56.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1ffff700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 0x7fcf200f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:56.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 -- 172.21.15.71:0/1317860269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 msgr2=0x7fcf200f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:56.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 0x7fcf200f9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 -- 172.21.15.71:0/1317860269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf14005040 con 0x7fcf200f6830 2021-11-20T07:40:56.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1f7fe700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcf14009f50 tx=0x7fcf14009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:56.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf14018070 con 0x7fcf200f6830 2021-11-20T07:40:56.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf14013410 con 0x7fcf200f6830 2021-11-20T07:40:56.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf200f8aa0 con 0x7fcf200f6830 2021-11-20T07:40:56.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1401c3c0 con 0x7fcf200f6830 2021-11-20T07:40:56.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.386+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf200f9000 con 0x7fcf200f6830 2021-11-20T07:40:56.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.387+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf20046330 con 0x7fcf200f6830 2021-11-20T07:40:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.387+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcf1401d420 con 0x7fcf200f6830 2021-11-20T07:40:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.388+0000 7fcf1cff9700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 0x7fcf0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.388+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fcf14077ec0 con 0x7fcf200f6830 2021-11-20T07:40:56.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.388+0000 7fcf1ffff700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 0x7fcf0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:56.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.388+0000 7fcf1ffff700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 0x7fcf0805e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fcf200f6690 tx=0x7fcf1000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:56.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.392+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf1404a880 con 0x7fcf200f6830 2021-11-20T07:40:56.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:56 smithi071 conmon[33687]: cluster 2021-11-20T07:40:56.399905+0000 mgr.smithi071.ynqmwn (mgr.14162) 269 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:56.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.579+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf200fdd20 con 0x7fcf200f6830 2021-11-20T07:40:56.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.579+0000 7fcf1cff9700 1 -- 172.21.15.71:0/1317860269 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fcf1404dea0 con 0x7fcf200f6830 2021-11-20T07:40:56.582 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:56.582 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:56.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 msgr2=0x7fcf0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:56.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 0x7fcf0805e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fcf200f6690 tx=0x7fcf1000b040).stop 2021-11-20T07:40:56.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 msgr2=0x7fcf200f8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:56.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcf14009f50 tx=0x7fcf14009b30).stop 2021-11-20T07:40:56.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 shutdown_connections 2021-11-20T07:40:56.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf200f5590 0x7fcf200f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf0805c2a0 0x7fcf0805e760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 --2- 172.21.15.71:0/1317860269 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf200f6830 0x7fcf200f8390 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:56.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 >> 172.21.15.71:0/1317860269 conn(0x7fcf200f0cd0 msgr2=0x7fcf2005fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:56.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 shutdown_connections 2021-11-20T07:40:56.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:56.581+0000 7fcf26679700 1 -- 172.21.15.71:0/1317860269 wait complete. 2021-11-20T07:40:56.586 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:56.595 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:56.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:56 smithi112 conmon[36062]: cluster 2021-11-20T07:40:56.399905+0000 mgr.smithi071.ynqmwn (mgr.14162) 269 : cluster 2021-11-20T07:40:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:56 smithi112 conmon[36062]: [DBG] pgmap v239: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:57.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:57.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- 172.21.15.71:0/227558475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f6b70 msgr2=0x7fb0fc0f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:57.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 --2- 172.21.15.71:0/227558475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc0f6f90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fb0f8004660 tx=0x7fb0f8009b30).stop 2021-11-20T07:40:57.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- 172.21.15.71:0/227558475 shutdown_connections 2021-11-20T07:40:57.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 --2- 172.21.15.71:0/227558475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 --2- 172.21.15.71:0/227558475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc0f6f90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- 172.21.15.71:0/227558475 >> 172.21.15.71:0/227558475 conn(0x7fb0fc0f0440 msgr2=0x7fb0fc0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:57.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- 172.21.15.71:0/227558475 shutdown_connections 2021-11-20T07:40:57.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- 172.21.15.71:0/227558475 wait complete. 2021-11-20T07:40:57.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 Processor -- start 2021-11-20T07:40:57.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- start start 2021-11-20T07:40:57.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:57.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:57.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc10c1a0 con 0x7fb0fc0f5870 2021-11-20T07:40:57.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.721+0000 7fb103736700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc10c2e0 con 0x7fb0fc0f6b70 2021-11-20T07:40:57.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:57.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48634/0 (socket says 172.21.15.71:48634) 2021-11-20T07:40:57.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 -- 172.21.15.71:0/91394339 learned_addr learned my addr 172.21.15.71:0/91394339 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:57.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb100cd1700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:57.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 -- 172.21.15.71:0/91394339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f6b70 msgr2=0x7fb0fc10bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:57.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc10bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 -- 172.21.15.71:0/91394339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f8005040 con 0x7fb0fc0f5870 2021-11-20T07:40:57.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb1014d2700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb0f8009f50 tx=0x7fb0f8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:57.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f8018070 con 0x7fb0fc0f5870 2021-11-20T07:40:57.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0f8013410 con 0x7fb0fc0f5870 2021-11-20T07:40:57.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc10c4e0 con 0x7fb0fc0f5870 2021-11-20T07:40:57.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0f801de70 con 0x7fb0fc0f5870 2021-11-20T07:40:57.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.722+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc10ca40 con 0x7fb0fc0f5870 2021-11-20T07:40:57.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.723+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc027a00 con 0x7fb0fc0f5870 2021-11-20T07:40:57.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.724+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb0f801d420 con 0x7fb0fc0f5870 2021-11-20T07:40:57.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.724+0000 7fb0f1ffb700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 0x7fb0e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:57.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.724+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb0f8077e10 con 0x7fb0fc0f5870 2021-11-20T07:40:57.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.724+0000 7fb100cd1700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 0x7fb0e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:57.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.725+0000 7fb100cd1700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 0x7fb0e805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb0fc0f6940 tx=0x7fb0ec00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:57.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.728+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0f804a7d0 con 0x7fb0fc0f5870 2021-11-20T07:40:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:57 smithi112 conmon[36062]: audit 2021-11-20T07:40:56.580928+0000 mon.smithi071 (mon.0) 2021-11-20T07:40:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:57 smithi112 conmon[36062]: 691 : audit [DBG] from='client.? 172.21.15.71:0/1317860269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:57.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:57 smithi071 conmon[33687]: audit 2021-11-20T07:40:56.580928+0000 mon.smithi071 (mon.0) 691 : audit [DBG] from='client.? 172.21.15.71:0/1317860269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:57.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.911+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0fc0fdd60 con 0x7fb0fc0f5870 2021-11-20T07:40:57.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.912+0000 7fb0f1ffb700 1 -- 172.21.15.71:0/91394339 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fb0f804ddf0 con 0x7fb0fc0f5870 2021-11-20T07:40:57.914 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:57.914 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:57.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 msgr2=0x7fb0e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:57.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 0x7fb0e805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb0fc0f6940 tx=0x7fb0ec00b040).stop 2021-11-20T07:40:57.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 msgr2=0x7fb0fc0f5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:57.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb0f8009f50 tx=0x7fb0f8009b30).stop 2021-11-20T07:40:57.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 shutdown_connections 2021-11-20T07:40:57.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb0e805c2a0 0x7fb0e805e760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0fc0f5870 0x7fb0fc0f5230 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 --2- 172.21.15.71:0/91394339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0fc0f6b70 0x7fb0fc10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:57.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 >> 172.21.15.71:0/91394339 conn(0x7fb0fc0f0440 msgr2=0x7fb0fc0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:57.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 shutdown_connections 2021-11-20T07:40:57.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:57.913+0000 7fb103736700 1 -- 172.21.15.71:0/91394339 wait complete. 2021-11-20T07:40:57.918 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:57.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:57.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:58 smithi112 conmon[36062]: audit 2021-11-20T07:40:57.913095+0000 mon.smithi071 (mon.0 2021-11-20T07:40:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:58 smithi112 conmon[36062]: ) 692 : audit [DBG] from='client.? 172.21.15.71:0/91394339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:58.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:58 smithi112 conmon[36062]: cluster 2021-11-20T07:40:58.400403+0000 mgr.smithi071.ynqmwn (mgr.14162) 270 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:58 smithi071 conmon[33687]: audit 2021-11-20T07:40:57.913095+0000 mon.smithi071 (mon.0) 692 : 2021-11-20T07:40:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:58 smithi071 conmon[33687]: audit [DBG] from='client.? 172.21.15.71:0/91394339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:58 smithi071 conmon[33687]: cluster 2021-11-20T07:40:58.400403+0000 mgr.smithi071.ynqmwn (mgr.14162) 270 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:40:58.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:40:59.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 -- 172.21.15.71:0/2538269676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c060100 msgr2=0x7f560c0f92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:59.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 --2- 172.21.15.71:0/2538269676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c060100 0x7f560c0f92c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f55fc0099e0 tx=0x7f55fc004d10).stop 2021-11-20T07:40:59.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 -- 172.21.15.71:0/2538269676 shutdown_connections 2021-11-20T07:40:59.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 --2- 172.21.15.71:0/2538269676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c0f9840 0x7f560c0fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:40:59.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 --2- 172.21.15.71:0/2538269676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c060100 0x7f560c0f92c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:59.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 -- 172.21.15.71:0/2538269676 >> 172.21.15.71:0/2538269676 conn(0x7f560c0f0ce0 msgr2=0x7f560c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:59.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 -- 172.21.15.71:0/2538269676 shutdown_connections 2021-11-20T07:40:59.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.052+0000 7f56117d9700 1 -- 172.21.15.71:0/2538269676 wait complete. 2021-11-20T07:40:59.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 Processor -- start 2021-11-20T07:40:59.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 -- start start 2021-11-20T07:40:59.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 0x7f560c107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:59.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:59.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c107b40 con 0x7f560c0f9840 2021-11-20T07:40:59.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c107c80 con 0x7f560c060100 2021-11-20T07:40:59.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48644/0 (socket says 172.21.15.71:48644) 2021-11-20T07:40:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 -- 172.21.15.71:0/3904620628 learned_addr learned my addr 172.21.15.71:0/3904620628 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:40:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560affd700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 0x7f560c107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 -- 172.21.15.71:0/3904620628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 msgr2=0x7f560c107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 0x7f560c107010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:59.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 -- 172.21.15.71:0/3904620628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc009640 con 0x7f560c0f9840 2021-11-20T07:40:59.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560affd700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 0x7f560c107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:40:59.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f560a7fc700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f55f40093f0 tx=0x7f55f4006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:59.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55f4013640 con 0x7f560c0f9840 2021-11-20T07:40:59.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55f4009e40 con 0x7f560c0f9840 2021-11-20T07:40:59.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c10c720 con 0x7f560c0f9840 2021-11-20T07:40:59.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.053+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55f400dbf0 con 0x7f560c0f9840 2021-11-20T07:40:59.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.054+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c10cc60 con 0x7f560c0f9840 2021-11-20T07:40:59.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.054+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c027a00 con 0x7f560c0f9840 2021-11-20T07:40:59.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.055+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f55f401a060 con 0x7f560c0f9840 2021-11-20T07:40:59.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.055+0000 7f56037fe700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:40:59.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.055+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f55f4076210 con 0x7f560c0f9840 2021-11-20T07:40:59.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.055+0000 7f560affd700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:40:59.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.056+0000 7f560affd700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 0x7f55f805e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f55fc00b620 tx=0x7f55fc004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:40:59.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.059+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55f4048b70 con 0x7f560c0f9840 2021-11-20T07:40:59.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.243+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f560c0feed0 con 0x7f560c0f9840 2021-11-20T07:40:59.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.244+0000 7f56037fe700 1 -- 172.21.15.71:0/3904620628 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f55f404c190 con 0x7f560c0f9840 2021-11-20T07:40:59.246 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:40:59.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:40:49.426437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14570},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14570":{"gid":14570,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":45,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6827/1128459258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1128459258},{"type":"v1","addr":"172.21.15.71:6827","nonce":1128459258}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:40:59.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 msgr2=0x7f55f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:59.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 0x7f55f805e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f55fc00b620 tx=0x7f55fc004d10).stop 2021-11-20T07:40:59.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 msgr2=0x7f560c107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:40:59.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f55f40093f0 tx=0x7f55f4006ec0).stop 2021-11-20T07:40:59.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 shutdown_connections 2021-11-20T07:40:59.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f560c060100 0x7f560c107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:59.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 --2- 172.21.15.71:0/3904620628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f560c0f9840 0x7f560c107500 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:40:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 >> 172.21.15.71:0/3904620628 conn(0x7f560c0f0ce0 msgr2=0x7f560c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:40:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.245+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 shutdown_connections 2021-11-20T07:40:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:40:59.246+0000 7f56117d9700 1 -- 172.21.15.71:0/3904620628 wait complete. 2021-11-20T07:40:59.251 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 53 2021-11-20T07:40:59.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:40:49.426437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14570}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14570': {'gid': 14570, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 45, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6827/1128459258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1128459258}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1128459258}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:40:59.260 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:resolve 2021-11-20T07:40:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:59 smithi071 conmon[33687]: audit 2021-11-20T07:40:59. 2021-11-20T07:40:59.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:40:59 smithi071 conmon[33687]: 245281+0000 mon.smithi071 (mon.0) 693 : audit [DBG] from='client.? 172.21.15.71:0/3904620628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:40:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:59 smithi112 conmon[36062]: audit 2021-11-20T07:40:59.245281+0000 mon.smithi071 (mon. 2021-11-20T07:40:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:40:59 smithi112 conmon[36062]: 0) 693 : audit [DBG] from='client.? 172.21.15.71:0/3904620628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:00.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:00.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 -- 172.21.15.71:0/1443960800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f6680 msgr2=0x7f49680f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:00.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 --2- 172.21.15.71:0/1443960800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f6680 0x7f49680f6aa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4958004660 tx=0x7f4958009b30).stop 2021-11-20T07:41:00.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 -- 172.21.15.71:0/1443960800 shutdown_connections 2021-11-20T07:41:00.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 --2- 172.21.15.71:0/1443960800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49680f7920 0x7f49680f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:00.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 --2- 172.21.15.71:0/1443960800 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f6680 0x7f49680f6aa0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:00.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 -- 172.21.15.71:0/1443960800 >> 172.21.15.71:0/1443960800 conn(0x7f49680f1de0 msgr2=0x7f49680f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:00.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 -- 172.21.15.71:0/1443960800 shutdown_connections 2021-11-20T07:41:00.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.375+0000 7f496d536700 1 -- 172.21.15.71:0/1443960800 wait complete. 2021-11-20T07:41:00.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 Processor -- start 2021-11-20T07:41:00.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 -- start start 2021-11-20T07:41:00.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49680f6680 0x7f4968106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:00.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:00.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968107a10 con 0x7f49680f7920 2021-11-20T07:41:00.384 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f496d536700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968107b50 con 0x7f49680f6680 2021-11-20T07:41:00.385 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f49667fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:00.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f49667fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48662/0 (socket says 172.21.15.71:48662) 2021-11-20T07:41:00.386 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.376+0000 7f49667fc700 1 -- 172.21.15.71:0/1908521281 learned_addr learned my addr 172.21.15.71:0/1908521281 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:00.387 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f49667fc700 1 -- 172.21.15.71:0/1908521281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49680f6680 msgr2=0x7f4968106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:00.387 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f49667fc700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49680f6680 0x7f4968106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:00.387 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f49667fc700 1 -- 172.21.15.71:0/1908521281 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958005040 con 0x7f49680f7920 2021-11-20T07:41:00.389 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f49667fc700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f495c00b410 tx=0x7f495c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:00.389 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495c0125f0 con 0x7f49680f7920 2021-11-20T07:41:00.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f495c00be80 con 0x7f49680f7920 2021-11-20T07:41:00.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496810c5f0 con 0x7f49680f7920 2021-11-20T07:41:00.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495c002ac0 con 0x7f49680f7920 2021-11-20T07:41:00.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.377+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496810cbb0 con 0x7f49680f7920 2021-11-20T07:41:00.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.378+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4968027a00 con 0x7f49680f7920 2021-11-20T07:41:00.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.378+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f495c00f030 con 0x7f49680f7920 2021-11-20T07:41:00.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.378+0000 7f494f7fe700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 0x7f495005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:00.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.378+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f495c0767c0 con 0x7f49680f7920 2021-11-20T07:41:00.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.379+0000 7f4966ffd700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 0x7f495005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:00.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.379+0000 7f4966ffd700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 0x7f495005e7b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4958009f50 tx=0x7f4958009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:00.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.382+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f495c049000 con 0x7f49680f7920 2021-11-20T07:41:00.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41:00.066733+0000 mon.smithi071 2021-11-20T07:41:00.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: (mon.0) 694 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:41:00.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41:00.070324+0000 mon.smithi071 (mon.0) 695 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T07:41:00.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41:00.395462+0000 mon.smithi071 (mon.0) 696 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 1 2021-11-20T07:41:00.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41:00.398778+0000 mon.smithi071 (mon.0) 697 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:boot 2021-11-20T07:41:00.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41:00. 2021-11-20T07:41:00.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: 398825+0000 mon.smithi071 (mon.0) 698 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:replay} 2 up:standby 2021-11-20T07:41:00.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: audit 2021-11-20T07:41:00.399102+0000 2021-11-20T07:41:00.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: mon.smithi071 (mon.0) 699 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:41:00.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster 2021-11-20T07:41 2021-11-20T07:41:00.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: :00.400674+0000 mgr.smithi071.ynqmwn (mgr.14162) 271 : 2021-11-20T07:41:00.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:00 smithi071 conmon[33687]: cluster [DBG] pgmap v242: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:41:00.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.576+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49680fee90 con 0x7f49680f7920 2021-11-20T07:41:00.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.576+0000 7f494f7fe700 1 -- 172.21.15.71:0/1908521281 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f495c04c620 con 0x7f49680f7920 2021-11-20T07:41:00.579 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:00.579 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:00.395517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:replay","state_seq":2,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:00.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 msgr2=0x7f495005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:00.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 0x7f495005e7b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4958009f50 tx=0x7f4958009b30).stop 2021-11-20T07:41:00.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 msgr2=0x7f49681073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:00.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f495c00b410 tx=0x7f495c006ef0).stop 2021-11-20T07:41:00.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 shutdown_connections 2021-11-20T07:41:00.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49680f6680 0x7f4968106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:00.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f495005c2f0 0x7f495005e7b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:00.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 --2- 172.21.15.71:0/1908521281 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49680f7920 0x7f49681073d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:00.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 >> 172.21.15.71:0/1908521281 conn(0x7f49680f1de0 msgr2=0x7f49680fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:00.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 shutdown_connections 2021-11-20T07:41:00.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:00.578+0000 7f496d536700 1 -- 172.21.15.71:0/1908521281 wait complete. 2021-11-20T07:41:00.583 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 54 2021-11-20T07:41:00.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:00.395517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:00.592 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14614:up:replay 2021-11-20T07:41:00.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11-20T07:41:00.066733+0000 mon.smithi071 (mon.0) 694 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:41:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11-20T07:41:00.070324+0000 mon.smithi071 (mon.0) 695 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T07:41:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11-20T07:41:00.395462+0000 mon.smithi071 (mon.0) 696 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 1 2021-11-20T07:41:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11-20T07:41:00.398778+0000 mon.smithi071 (mon 2021-11-20T07:41:00.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: .0) 697 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:boot 2021-11-20T07:41:00.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:41:00.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: 07:41:00.398825+0000 mon.smithi071 (mon.0) 698 2021-11-20T07:41:00.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:replay} 2 up:standby 2021-11-20T07:41:00.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: audit 2021-11-20T07:41:00.399102 2021-11-20T07:41:00.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 699 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:41:00.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:41:00.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: 20T07:41:00.400674+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:41:00.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:00 smithi112 conmon[36062]: 14162) 271 : cluster [DBG] pgmap v242: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:41:01.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:01.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- 172.21.15.71:0/199730971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f6840 msgr2=0x7f6f300f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:01.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 --2- 172.21.15.71:0/199730971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f6840 0x7f6f300f6ca0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6f20004660 tx=0x7f6f20009b30).stop 2021-11-20T07:41:01.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- 172.21.15.71:0/199730971 shutdown_connections 2021-11-20T07:41:01.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 --2- 172.21.15.71:0/199730971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f6840 0x7f6f300f6ca0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 --2- 172.21.15.71:0/199730971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f55a0 0x7f6f300f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- 172.21.15.71:0/199730971 >> 172.21.15.71:0/199730971 conn(0x7f6f300f0ce0 msgr2=0x7f6f300f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:01.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- 172.21.15.71:0/199730971 shutdown_connections 2021-11-20T07:41:01.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- 172.21.15.71:0/199730971 wait complete. 2021-11-20T07:41:01.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 Processor -- start 2021-11-20T07:41:01.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- start start 2021-11-20T07:41:01.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:01.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f6840 0x7f6f3010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:01.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f30107a90 con 0x7f6f300f55a0 2021-11-20T07:41:01.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3749f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3010e5a0 con 0x7f6f300f6840 2021-11-20T07:41:01.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.719+0000 7f6f3523b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:01.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48684/0 (socket says 172.21.15.71:48684) 2021-11-20T07:41:01.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 -- 172.21.15.71:0/3973735851 learned_addr learned my addr 172.21.15.71:0/3973735851 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:01.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f34a3a700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f6840 0x7f6f3010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:01.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 -- 172.21.15.71:0/3973735851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f6840 msgr2=0x7f6f3010e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:01.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f6840 0x7f6f3010e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 -- 172.21.15.71:0/3973735851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f20005040 con 0x7f6f300f55a0 2021-11-20T07:41:01.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3523b700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6f2c002700 tx=0x7f6f2c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:01.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f2c0079c0 con 0x7f6f300f55a0 2021-11-20T07:41:01.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f2c00f040 con 0x7f6f300f55a0 2021-11-20T07:41:01.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3010e740 con 0x7f6f300f55a0 2021-11-20T07:41:01.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f2c014860 con 0x7f6f300f55a0 2021-11-20T07:41:01.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.720+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3010ecd0 con 0x7f6f300f55a0 2021-11-20T07:41:01.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.721+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f30046330 con 0x7f6f300f55a0 2021-11-20T07:41:01.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.721+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6f2c01b060 con 0x7f6f300f55a0 2021-11-20T07:41:01.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.722+0000 7f6f25ffb700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 0x7f6f1c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:01.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.722+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f6f2c076070 con 0x7f6f300f55a0 2021-11-20T07:41:01.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.722+0000 7f6f34a3a700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 0x7f6f1c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:01.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.722+0000 7f6f34a3a700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 0x7f6f1c05e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6f20009f50 tx=0x7f6f20009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:01.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.726+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f2c0489d0 con 0x7f6f300f55a0 2021-11-20T07:41:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:01 smithi112 conmon[36062]: audit 2021-11-20T07:41:00.577561+0000 mon.smithi071 (mon. 2021-11-20T07:41:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:01 smithi112 conmon[36062]: 0) 700 : audit [DBG] from='client.? 172.21.15.71:0/1908521281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:01.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:01 smithi112 conmon[36062]: cluster 2021-11-20T07:41:01.409086+0000 mon.smithi071 (mon.0) 701 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:resolve 2021-11-20T07:41:01.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:01 smithi112 conmon[36062]: cluster 2021-11-20T07:41:01.409152+0000 mon.smithi071 (mon.0) 702 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:resolve} 2 up:standby 2021-11-20T07:41:01.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:01 smithi112 conmon[36062]: 2021-11-20T07:41:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:01 smithi071 conmon[33687]: audit 2021-11-20T07:41:00.577561+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:01 smithi071 conmon[33687]: 700 : audit [DBG] from='client.? 172.21.15.71:0/1908521281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:01.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:01 smithi071 conmon[33687]: cluster 2021-11-20T07:41:01.409086+0000 mon.smithi071 (mon.0) 701 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:resolve 2021-11-20T07:41:01.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:01 smithi071 conmon[33687]: cluster 2021-11-20T07:41:01.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:01 smithi071 conmon[33687]: 2021-11-20T07:41:01.409152+0000 mon.smithi071 (mon.0) 702 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.zaejme=up:resolve} 2 up:standby 2021-11-20T07:41:01.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.912+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f30027a00 con 0x7f6f300f55a0 2021-11-20T07:41:01.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.912+0000 7f6f25ffb700 1 -- 172.21.15.71:0/3973735851 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f6f2c04bff0 con 0x7f6f300f55a0 2021-11-20T07:41:01.915 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:01.915 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:01.400425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:resolve","state_seq":5,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:resolve","state_seq":16,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:01.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 msgr2=0x7f6f1c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:01.916 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 0x7f6f1c05e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6f20009f50 tx=0x7f6f20009b30).stop 2021-11-20T07:41:01.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 msgr2=0x7f6f3005ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:01.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6f2c002700 tx=0x7f6f2c009d80).stop 2021-11-20T07:41:01.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 shutdown_connections 2021-11-20T07:41:01.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6f1c05c2a0 0x7f6f1c05e760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f300f55a0 0x7f6f3005ff00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 --2- 172.21.15.71:0/3973735851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f300f6840 0x7f6f3010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:01.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 >> 172.21.15.71:0/3973735851 conn(0x7f6f300f0ce0 msgr2=0x7f6f300fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:01.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 shutdown_connections 2021-11-20T07:41:01.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:01.914+0000 7f6f3749f700 1 -- 172.21.15.71:0/3973735851 wait complete. 2021-11-20T07:41:01.919 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 55 2021-11-20T07:41:01.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:01.400425+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:resolve', 'state_seq': 5, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:01.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14614:up:resolve 2021-11-20T07:41:02.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: audit 2021-11-20T07:41:01.913814+0000 mon.smithi071 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.71:0/3973735851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:02.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: 2021-11-20T07:41:02.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: cluster 2021-11-20T07:41:02.401355+0000 mgr.smithi071.ynqmwn (mgr.14162) 272 : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:41:02.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: cluster 2021-11-20T07:41:02.410449+0000 2021-11-20T07:41:02.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: mon.smithi071 (mon.0) 704 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:reconnect 2021-11-20T07:41:02.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:41:02.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: 11-20T07:41:02.410496+0000 mon.smithi071 (mon.0) 705 2021-11-20T07:41:02.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:reconnect 2021-11-20T07:41:02.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:41:02.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: T07:41:02.410534+0000 mon.smithi071 (mon. 2021-11-20T07:41:02.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:02 smithi112 conmon[36062]: 0) 706 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi071.zaejme=up:reconnect} 2 up:standby 2021-11-20T07:41:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: audit 2021-11-20T07:41:01.913814+0000 mon.smithi071 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.71:0/3973735851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: cluster 2021-11-20T07:41:02.401355+0000 mgr.smithi071.ynqmwn (mgr.14162) 272 : cluster 2021-11-20T07:41:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: [DBG] pgmap v243: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:41:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: cluster 2021-11-20T07:41:02.410449+0000 mon.smithi071 (mon.0) 704 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:reconnect 2021-11-20T07:41:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: cluster 2021-11-20T07:41:02.410496+0000 2021-11-20T07:41:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: mon.smithi071 (mon.0) 705 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:reconnect 2021-11-20T07:41:02.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:41:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: 11-20T07:41:02.410534+0000 mon.smithi071 (mon.0 2021-11-20T07:41:02.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:02 smithi071 conmon[33687]: ) 706 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi071.zaejme=up:reconnect} 2 up:standby 2021-11-20T07:41:02.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:03.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 -- 172.21.15.71:0/3810131851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 msgr2=0x7f7be40f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:03.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 --2- 172.21.15.71:0/3810131851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be40f6c00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f7bd4004660 tx=0x7f7bd4009b30).stop 2021-11-20T07:41:03.046 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 -- 172.21.15.71:0/3810131851 shutdown_connections 2021-11-20T07:41:03.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 --2- 172.21.15.71:0/3810131851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be40f6c00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:03.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 --2- 172.21.15.71:0/3810131851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 0x7f7be40f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:03.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 -- 172.21.15.71:0/3810131851 >> 172.21.15.71:0/3810131851 conn(0x7f7be40f0c40 msgr2=0x7f7be40f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:03.047 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 -- 172.21.15.71:0/3810131851 shutdown_connections 2021-11-20T07:41:03.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.042+0000 7f7beab1a700 1 -- 172.21.15.71:0/3810131851 wait complete. 2021-11-20T07:41:03.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 Processor -- start 2021-11-20T07:41:03.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 -- start start 2021-11-20T07:41:03.048 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 0x7f7be4109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:03.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:03.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be40605d0 con 0x7f7be40f67a0 2021-11-20T07:41:03.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be4060710 con 0x7f7be40f5500 2021-11-20T07:41:03.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:03.049 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48700/0 (socket says 172.21.15.71:48700) 2021-11-20T07:41:03.050 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 -- 172.21.15.71:0/1793327205 learned_addr learned my addr 172.21.15.71:0/1793327205 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:03.050 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9b18700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 0x7f7be4109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:03.050 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 -- 172.21.15.71:0/1793327205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 msgr2=0x7f7be4109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:03.050 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 0x7f7be4109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:03.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 -- 172.21.15.71:0/1793327205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4005040 con 0x7f7be40f67a0 2021-11-20T07:41:03.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7be9317700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f7bd4009830 tx=0x7f7bd4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:03.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd4018070 con 0x7f7be40f67a0 2021-11-20T07:41:03.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bd4013410 con 0x7f7be40f67a0 2021-11-20T07:41:03.051 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be410c530 con 0x7f7be40f67a0 2021-11-20T07:41:03.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.043+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd401c620 con 0x7f7be40f67a0 2021-11-20T07:41:03.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.044+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be410ca90 con 0x7f7be40f67a0 2021-11-20T07:41:03.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.044+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be4045ad0 con 0x7f7be40f67a0 2021-11-20T07:41:03.052 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.045+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7bd401c890 con 0x7f7be40f67a0 2021-11-20T07:41:03.053 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.045+0000 7f7bda7fc700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 0x7f7bd005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:03.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.045+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f7bd4077ff0 con 0x7f7be40f67a0 2021-11-20T07:41:03.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.045+0000 7f7be9b18700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 0x7f7bd005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:03.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.045+0000 7f7be9b18700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 0x7f7bd005e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f7be40601c0 tx=0x7f7be0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:03.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.049+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bd404a9d0 con 0x7f7be40f67a0 2021-11-20T07:41:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.231+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7be4027a00 con 0x7f7be40f67a0 2021-11-20T07:41:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.232+0000 7f7bda7fc700 1 -- 172.21.15.71:0/1793327205 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2956 (secure 0 0 0) 0x7f7bd404dff0 con 0x7f7be40f67a0 2021-11-20T07:41:03.234 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:03.235 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:02.406961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 msgr2=0x7f7bd005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 0x7f7bd005e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f7be40601c0 tx=0x7f7be0008040).stop 2021-11-20T07:41:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 msgr2=0x7f7be410bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f7bd4009830 tx=0x7f7bd4009b30).stop 2021-11-20T07:41:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 shutdown_connections 2021-11-20T07:41:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.233+0000 7f7beab1a700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be40f5500 0x7f7be4109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.234+0000 7f7beab1a700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7bd005c300 0x7f7bd005e7c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.234+0000 7f7beab1a700 1 --2- 172.21.15.71:0/1793327205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7be40f67a0 0x7f7be410bf80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.234+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 >> 172.21.15.71:0/1793327205 conn(0x7f7be40f0c40 msgr2=0x7f7be40fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.234+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 shutdown_connections 2021-11-20T07:41:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:03.234+0000 7f7beab1a700 1 -- 172.21.15.71:0/1793327205 wait complete. 2021-11-20T07:41:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 56 2021-11-20T07:41:03.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:02.406961+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:reconnect', 'state_seq': 20, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:03.247 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14614:up:reconnect 2021-11-20T07:41:03.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021-11-20T07:41:02.434233+0000 mds.cephfs.smithi071.zaejme (mds. 2021-11-20T07:41:03.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249996 2021-11-20T07:41:03.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021-11-20T07:41:02.434562+0000 mds.cephfs.smithi112.iqrmps (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259993 2021-11-20T07:41:03.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: audit 2021-11-20T07:41:03.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 2021-11-20T07:41:03.233334+0000 mon.smithi071 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.71:0/1793327205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:03.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 2021-11-20T07:41:03.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:41:03.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 11-20T07:41:03.411921+0000 mon.smithi071 (mon.0) 708 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:rejoin 2021-11-20T07:41:03.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 2021-11-20T07:41:03.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:41:03.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 20T07:41:03.411965+0000 mon.smithi071 (mon.0) 709 2021-11-20T07:41:03.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:rejoin 2021-11-20T07:41:03.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021-11-20T07:41 2021-11-20T07:41:03.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: :03.412032+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:03.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: 710 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi071.zaejme=up:rejoin} 2 up:standby 2021-11-20T07:41:03.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: cluster 2021-11-20T07:41:03 2021-11-20T07:41:03.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:03 smithi071 conmon[33687]: .414156+0000 mon.smithi071 (mon.0) 711 : cluster [INF] daemon mds.cephfs.smithi071.zaejme is now active in filesystem cephfs as rank 1 2021-11-20T07:41:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021-11-20T07:41:02.434233+0000 mds.cephfs.smithi071.zaejme (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249996 2021-11-20T07:41:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: 2021-11-20T07:41:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021-11-20T07:41:02.434562+0000 mds.cephfs.smithi112.iqrmps (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259993 2021-11-20T07:41:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: audit 2021-11-20T07:41:03.233334+0000 mon.smithi071 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.71:0/1793327205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021-11-20T07:41:03.411921+0000 mon.smithi071 (mon.0) 708 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:rejoin 2021-11-20T07:41:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021-11-20T07:41 2021-11-20T07:41:03.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: :03.411965+0000 mon.smithi071 (mon.0) 709 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:rejoin 2021-11-20T07:41:03.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:41:03.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: -11-20T07:41:03.412032+0000 mon.smithi071 (mon.0) 710 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi071.zaejme=up:rejoin} 2 up:standby 2021-11-20T07:41:03.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: cluster 2021-11-20T07:41:03.414156+0000 mon.smithi071 (mon 2021-11-20T07:41:03.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:03 smithi112 conmon[36062]: .0) 711 : cluster [INF] daemon mds.cephfs.smithi071.zaejme is now active in filesystem cephfs as rank 1 2021-11-20T07:41:04.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:04.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- 172.21.15.71:0/858201887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 msgr2=0x7f04b40f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:04.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 --2- 172.21.15.71:0/858201887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b40f59c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f04a40099e0 tx=0x7f04a4004d10).stop 2021-11-20T07:41:04.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- 172.21.15.71:0/858201887 shutdown_connections 2021-11-20T07:41:04.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 --2- 172.21.15.71:0/858201887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 0x7f04b40f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 --2- 172.21.15.71:0/858201887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b40f59c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- 172.21.15.71:0/858201887 >> 172.21.15.71:0/858201887 conn(0x7f04b40f0ce0 msgr2=0x7f04b40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:04.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- 172.21.15.71:0/858201887 shutdown_connections 2021-11-20T07:41:04.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- 172.21.15.71:0/858201887 wait complete. 2021-11-20T07:41:04.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 Processor -- start 2021-11-20T07:41:04.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.370+0000 7f04b8980700 1 -- start start 2021-11-20T07:41:04.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:04.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 0x7f04b410bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:04.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b410c290 con 0x7f04b40f55a0 2021-11-20T07:41:04.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b410c3d0 con 0x7f04b40f6840 2021-11-20T07:41:04.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:04.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48708/0 (socket says 172.21.15.71:48708) 2021-11-20T07:41:04.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 -- 172.21.15.71:0/1624670142 learned_addr learned my addr 172.21.15.71:0/1624670142 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:04.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b1d9b700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 0x7f04b410bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:04.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 -- 172.21.15.71:0/1624670142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 msgr2=0x7f04b410bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:04.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 0x7f04b410bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 -- 172.21.15.71:0/1624670142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4009640 con 0x7f04b40f55a0 2021-11-20T07:41:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b259c700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f04a40078e0 tx=0x7f04a4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a400e030 con 0x7f04b40f55a0 2021-11-20T07:41:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b410c5d0 con 0x7f04b40f55a0 2021-11-20T07:41:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04a4007d80 con 0x7f04b40f55a0 2021-11-20T07:41:04.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a4018070 con 0x7f04b40f55a0 2021-11-20T07:41:04.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.371+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b410cab0 con 0x7f04b40f55a0 2021-11-20T07:41:04.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b4027a00 con 0x7f04b40f55a0 2021-11-20T07:41:04.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f04a4024060 con 0x7f04b40f55a0 2021-11-20T07:41:04.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04aaffd700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 0x7f04a005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:04.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f04a40777d0 con 0x7f04b40f55a0 2021-11-20T07:41:04.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04b1d9b700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 0x7f04a005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:04.383 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.373+0000 7f04b1d9b700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 0x7f04a005e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f049c004c40 tx=0x7f049c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.377+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04a404a130 con 0x7f04b40f55a0 2021-11-20T07:41:04.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.583+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04b40fdd30 con 0x7f04b40f55a0 2021-11-20T07:41:04.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.584+0000 7f04aaffd700 1 -- 172.21.15.71:0/1624670142 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f04a404d750 con 0x7f04b40f55a0 2021-11-20T07:41:04.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:04.586 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:04.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 msgr2=0x7f04a005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:04.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 0x7f04a005e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f049c004c40 tx=0x7f049c00b040).stop 2021-11-20T07:41:04.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 msgr2=0x7f04b405ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:04.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f04a40078e0 tx=0x7f04a4004d10).stop 2021-11-20T07:41:04.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 shutdown_connections 2021-11-20T07:41:04.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f04a005c2a0 0x7f04a005e760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04b40f55a0 0x7f04b405ff20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 --2- 172.21.15.71:0/1624670142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04b40f6840 0x7f04b410bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:04.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 >> 172.21.15.71:0/1624670142 conn(0x7f04b40f0ce0 msgr2=0x7f04b40fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:04.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 shutdown_connections 2021-11-20T07:41:04.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:04.585+0000 7f04b8980700 1 -- 172.21.15.71:0/1624670142 wait complete. 2021-11-20T07:41:04.590 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:04.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:04.598 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:05.459 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: cluster 2021-11-20T07:41:04.401737+0000 mgr.smithi071.ynqmwn (mgr.14162) 273 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 5.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:41:05.459 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: cluster 2021-11-20T07:41:04.413019+0000 mon.smithi071 (mon.0) 712 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:active 2021-11-20T07:41:05.459 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: cluster 2021-11-20T07:41: 2021-11-20T07:41:05.460 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: 04.413069+0000 mon.smithi071 (mon.0) 713 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:clientreplay 2021-11-20T07:41:05.460 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:41:05.460 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: 20T07:41:04.413108+0000 2021-11-20T07:41:05.460 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: mon.smithi071 (mon.0) 714 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:standby 2021-11-20T07:41:05.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: 2021-11-20T07:41:05.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: cluster 2021-11-20T07:41:05.461 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: 2021-11-20T07:41:04.413163 2021-11-20T07:41:05.462 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 715 : cluster [DBG] 2021-11-20T07:41:05.462 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:05.462 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:41:05.462 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: 41:04.584998+0000 mon.smithi071 (mon.0 2021-11-20T07:41:05.463 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:05 smithi071 conmon[33687]: ) 716 : audit [DBG] from='client.? 172.21.15.71:0/1624670142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:05.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:05.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 -- 172.21.15.71:0/1940457836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f6290 msgr2=0x7fa4a40f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:05.720 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 --2- 172.21.15.71:0/1940457836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f6290 0x7fa4a40f66f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fa494004660 tx=0x7fa494009b30).stop 2021-11-20T07:41:05.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 -- 172.21.15.71:0/1940457836 shutdown_connections 2021-11-20T07:41:05.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 --2- 172.21.15.71:0/1940457836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f6290 0x7fa4a40f66f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:05.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 --2- 172.21.15.71:0/1940457836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f7c80 0x7fa4a40f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:05.721 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 -- 172.21.15.71:0/1940457836 >> 172.21.15.71:0/1940457836 conn(0x7fa4a40f1d30 msgr2=0x7fa4a40f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:05.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 -- 172.21.15.71:0/1940457836 shutdown_connections 2021-11-20T07:41:05.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.715+0000 7fa4aa262700 1 -- 172.21.15.71:0/1940457836 wait complete. 2021-11-20T07:41:05.722 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 Processor -- start 2021-11-20T07:41:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 -- start start 2021-11-20T07:41:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f6290 0x7fa4a406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a406a410 con 0x7fa4a40f7c80 2021-11-20T07:41:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4aa262700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a406a550 con 0x7fa4a40f6290 2021-11-20T07:41:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48722/0 (socket says 172.21.15.71:48722) 2021-11-20T07:41:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 -- 172.21.15.71:0/3296348000 learned_addr learned my addr 172.21.15.71:0/3296348000 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:05.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a37fe700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f6290 0x7fa4a406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:05.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 -- 172.21.15.71:0/3296348000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f6290 msgr2=0x7fa4a406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:05.725 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f6290 0x7fa4a406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:05.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.716+0000 7fa4a2ffd700 1 -- 172.21.15.71:0/3296348000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494005040 con 0x7fa4a40f7c80 2021-11-20T07:41:05.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa4a2ffd700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa494009830 tx=0x7fa494009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:05.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa494018070 con 0x7fa4a40f7c80 2021-11-20T07:41:05.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa494013410 con 0x7fa4a40f7c80 2021-11-20T07:41:05.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49401c620 con 0x7fa4a40f7c80 2021-11-20T07:41:05.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a406a7d0 con 0x7fa4a40f7c80 2021-11-20T07:41:05.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.717+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a410ef00 con 0x7fa4a40f7c80 2021-11-20T07:41:05.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.718+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4045ad0 con 0x7fa4a40f7c80 2021-11-20T07:41:05.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.718+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa49401c890 con 0x7fa4a40f7c80 2021-11-20T07:41:05.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.718+0000 7fa48bfff700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 0x7fa48c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:05.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.718+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fa494078050 con 0x7fa4a40f7c80 2021-11-20T07:41:05.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.718+0000 7fa4a37fe700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 0x7fa48c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:05.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.719+0000 7fa4a37fe700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 0x7fa48c05e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa4a406b730 tx=0x7fa498008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:05.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.722+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa49404a9d0 con 0x7fa4a40f7c80 2021-11-20T07:41:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: cluster 2021-11-20T07:41:04.401737+0000 mgr.smithi071.ynqmwn (mgr.14162) 273 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 5.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:41:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: cluster 2021-11-20T07:41:04.413019+0000 mon.smithi071 (mon.0) 712 : cluster [DBG] mds.? [v2:172.21.15.71:6828/741236990,v1:172.21.15.71:6829/741236990] up:active 2021-11-20T07:41:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: cluster 2021-11-20T07:41:04.413069+0000 mon.smithi071 (mon.0) 713 : cluster [DBG] mds.? [v2:172.21.15.112:6826/3888903196,v1:172.21.15.112:6827/3888903196] up:clientreplay 2021-11-20T07:41:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: cluster 2021-11-20T07:41:04.413108+0000 mon.smithi071 (mon.0) 714 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:standby 2021-11-20T07:41:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: cluster 2021-11-20T07:41:04.413163+0000 mon.smithi071 (mon.0) 715 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:41:05.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:05 smithi112 conmon[36062]: T07:41:04.584998+0000 mon.smithi071 (mon.0) 716 : audit [DBG] from='client.? 172.21.15.71:0/1624670142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:05.907 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.905+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a40feee0 con 0x7fa4a40f7c80 2021-11-20T07:41:05.908 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.906+0000 7fa48bfff700 1 -- 172.21.15.71:0/3296348000 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7fa49404dff0 con 0x7fa4a40f7c80 2021-11-20T07:41:05.908 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:05.908 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:05.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.907+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 msgr2=0x7fa48c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:05.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 0x7fa48c05e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa4a406b730 tx=0x7fa498008040).stop 2021-11-20T07:41:05.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 msgr2=0x7fa4a4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:05.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa494009830 tx=0x7fa494009b30).stop 2021-11-20T07:41:05.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 shutdown_connections 2021-11-20T07:41:05.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4a40f6290 0x7fa4a406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:05.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa48c05c300 0x7fa48c05e7c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:05.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 --2- 172.21.15.71:0/3296348000 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4a40f7c80 0x7fa4a4069e20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:05.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 >> 172.21.15.71:0/3296348000 conn(0x7fa4a40f1d30 msgr2=0x7fa4a40fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:05.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 shutdown_connections 2021-11-20T07:41:05.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:05.908+0000 7fa4aa262700 1 -- 172.21.15.71:0/3296348000 wait complete. 2021-11-20T07:41:05.912 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:05.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:05.922 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:06.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:06 smithi112 conmon[36062]: audit 2021-11-20T07:41:05.907240+0000 mon.smithi071 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.71:0/3296348000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:06.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:06 smithi071 conmon[33687]: audit 2021-11-20T07:41:05.907240+0000 mon.smithi071 (mon.0 2021-11-20T07:41:06.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:06 smithi071 conmon[33687]: ) 717 : audit [DBG] from='client.? 172.21.15.71:0/3296348000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:06.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:07.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 -- 172.21.15.71:0/535201498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f6850 msgr2=0x7f034c0f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:07.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 --2- 172.21.15.71:0/535201498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f6850 0x7f034c0f6cb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f0340004660 tx=0x7f0340009b30).stop 2021-11-20T07:41:07.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 -- 172.21.15.71:0/535201498 shutdown_connections 2021-11-20T07:41:07.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 --2- 172.21.15.71:0/535201498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f6850 0x7f034c0f6cb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:07.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 --2- 172.21.15.71:0/535201498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f55b0 0x7f034c0f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 -- 172.21.15.71:0/535201498 >> 172.21.15.71:0/535201498 conn(0x7f034c0f0d10 msgr2=0x7f034c0f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 -- 172.21.15.71:0/535201498 shutdown_connections 2021-11-20T07:41:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.047+0000 7f0352057700 1 -- 172.21.15.71:0/535201498 wait complete. 2021-11-20T07:41:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 Processor -- start 2021-11-20T07:41:07.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 -- start start 2021-11-20T07:41:07.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:07.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 0x7f034c10bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:07.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c10c1d0 con 0x7f034c0f55b0 2021-11-20T07:41:07.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f0352057700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c10c310 con 0x7f034c0f6850 2021-11-20T07:41:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f034b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f034b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48734/0 (socket says 172.21.15.71:48734) 2021-11-20T07:41:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f034b7fe700 1 -- 172.21.15.71:0/1336466108 learned_addr learned my addr 172.21.15.71:0/1336466108 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.048+0000 7f034affd700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 0x7f034c10bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f034b7fe700 1 -- 172.21.15.71:0/1336466108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 msgr2=0x7f034c10bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:07.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f034b7fe700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 0x7f034c10bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:07.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f034b7fe700 1 -- 172.21.15.71:0/1336466108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0340005040 con 0x7f034c0f55b0 2021-11-20T07:41:07.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f034affd700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 0x7f034c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:41:07.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f034b7fe700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f033c00b790 tx=0x7f033c0092b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:07.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033c00d700 con 0x7f034c0f55b0 2021-11-20T07:41:07.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f033c00dd10 con 0x7f034c0f55b0 2021-11-20T07:41:07.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c10c5a0 con 0x7f034c0f55b0 2021-11-20T07:41:07.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033c010e10 con 0x7f034c0f55b0 2021-11-20T07:41:07.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.049+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c10cb60 con 0x7f034c0f55b0 2021-11-20T07:41:07.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.050+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c027a00 con 0x7f034c0f55b0 2021-11-20T07:41:07.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.050+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f033c0105e0 con 0x7f034c0f55b0 2021-11-20T07:41:07.061 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.050+0000 7f0333fff700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 0x7f033405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:07.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.050+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f033c0764e0 con 0x7f034c0f55b0 2021-11-20T07:41:07.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.050+0000 7f034affd700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 0x7f033405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:07.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.051+0000 7f034affd700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 0x7f033405e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0340013580 tx=0x7f0340009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:07.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.054+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f033c048e40 con 0x7f034c0f55b0 2021-11-20T07:41:07.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.243+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034c0fdde0 con 0x7f034c0f55b0 2021-11-20T07:41:07.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.243+0000 7f0333fff700 1 -- 172.21.15.71:0/1336466108 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f033c016090 con 0x7f034c0f55b0 2021-11-20T07:41:07.246 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:07.246 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:07.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 msgr2=0x7f033405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:07.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 0x7f033405e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0340013580 tx=0x7f0340009b30).stop 2021-11-20T07:41:07.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 msgr2=0x7f034c060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:07.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f033c00b790 tx=0x7f033c0092b0).stop 2021-11-20T07:41:07.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 shutdown_connections 2021-11-20T07:41:07.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f033405c2a0 0x7f033405e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:07.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f034c0f55b0 0x7f034c060510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:07.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 --2- 172.21.15.71:0/1336466108 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f034c0f6850 0x7f034c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:07.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 >> 172.21.15.71:0/1336466108 conn(0x7f034c0f0d10 msgr2=0x7f034c0fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:07.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 shutdown_connections 2021-11-20T07:41:07.251 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:07.245+0000 7f0352057700 1 -- 172.21.15.71:0/1336466108 wait complete. 2021-11-20T07:41:07.251 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:07.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:07.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:07 smithi071 conmon[33687]: cluster 2021-11-20T07:41:06.402095+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:41:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:07 smithi071 conmon[33687]: 274 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:41:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:07 smithi071 conmon[33687]: audit 2021-11-20T07:41:07.244708+0000 mon.smithi071 (mon.0) 718 2021-11-20T07:41:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:07 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1336466108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:07 smithi112 conmon[36062]: cluster 2021-11-20T07:41:06.402095+0000 mgr.smithi071.ynqmwn (mgr.14162) 274 : cluster 2021-11-20T07:41:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:07 smithi112 conmon[36062]: [DBG] pgmap v245: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:41:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:07 smithi112 conmon[36062]: audit 2021-11-20T07:41:07.244708+0000 mon.smithi071 (mon.0) 718 : audit [DBG] from='client.? 172.21.15.71:0/1336466108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:08.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:08.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- 172.21.15.71:0/3524177037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 msgr2=0x7f4db40f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:08.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 --2- 172.21.15.71:0/3524177037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db40f5e50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f4da8004660 tx=0x7f4da8009b30).stop 2021-11-20T07:41:08.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- 172.21.15.71:0/3524177037 shutdown_connections 2021-11-20T07:41:08.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 --2- 172.21.15.71:0/3524177037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db40f5e50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 --2- 172.21.15.71:0/3524177037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 0x7f4db40f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- 172.21.15.71:0/3524177037 >> 172.21.15.71:0/3524177037 conn(0x7f4db40f04a0 msgr2=0x7f4db40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:08.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- 172.21.15.71:0/3524177037 shutdown_connections 2021-11-20T07:41:08.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- 172.21.15.71:0/3524177037 wait complete. 2021-11-20T07:41:08.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 Processor -- start 2021-11-20T07:41:08.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- start start 2021-11-20T07:41:08.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:08.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 0x7f4db4106340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:08.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41068f0 con 0x7f4db40f59d0 2021-11-20T07:41:08.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db8f53700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db4106a30 con 0x7f4db40f6d10 2021-11-20T07:41:08.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:08.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db259c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48746/0 (socket says 172.21.15.71:48746) 2021-11-20T07:41:08.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.389+0000 7f4db259c700 1 -- 172.21.15.71:0/7541728 learned_addr learned my addr 172.21.15.71:0/7541728 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:08.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db1d9b700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 0x7f4db4106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:08.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db259c700 1 -- 172.21.15.71:0/7541728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 msgr2=0x7f4db4106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:08.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db259c700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 0x7f4db4106340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db259c700 1 -- 172.21.15.71:0/7541728 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da8005040 con 0x7f4db40f59d0 2021-11-20T07:41:08.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db259c700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f4da40026d0 tx=0x7f4da4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:08.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da4009320 con 0x7f4db40f59d0 2021-11-20T07:41:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4da400f040 con 0x7f4db40f59d0 2021-11-20T07:41:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db410b4d0 con 0x7f4db40f59d0 2021-11-20T07:41:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da40147e0 con 0x7f4db40f59d0 2021-11-20T07:41:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.390+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db410ba90 con 0x7f4db40f59d0 2021-11-20T07:41:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.391+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db4027a00 con 0x7f4db40f59d0 2021-11-20T07:41:08.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.391+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4da4009480 con 0x7f4db40f59d0 2021-11-20T07:41:08.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.391+0000 7f4da2ffd700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 0x7f4d9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:08.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.392+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f4da4076960 con 0x7f4db40f59d0 2021-11-20T07:41:08.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.392+0000 7f4db1d9b700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 0x7f4d9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:08.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.392+0000 7f4db1d9b700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 0x7f4d9c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4da8004e30 tx=0x7f4da8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:08.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.395+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4da40492c0 con 0x7f4db40f59d0 2021-11-20T07:41:08.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:08 smithi071 conmon[33687]: cluster 2021-11-20T07:41:08.402465+0000 2021-11-20T07:41:08.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:08 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 275 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:08.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.583+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4db40fded0 con 0x7f4db40f59d0 2021-11-20T07:41:08.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.584+0000 7f4da2ffd700 1 -- 172.21.15.71:0/7541728 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f4da404c8e0 con 0x7f4db40f59d0 2021-11-20T07:41:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:08.586 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:08.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 msgr2=0x7f4d9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:08.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 0x7f4d9c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4da8004e30 tx=0x7f4da8009b30).stop 2021-11-20T07:41:08.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 msgr2=0x7f4db4105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:08.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f4da40026d0 tx=0x7f4da4009de0).stop 2021-11-20T07:41:08.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 shutdown_connections 2021-11-20T07:41:08.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4d9c05c2a0 0x7f4d9c05e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db40f59d0 0x7f4db4105e50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 --2- 172.21.15.71:0/7541728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4db40f6d10 0x7f4db4106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:08.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 >> 172.21.15.71:0/7541728 conn(0x7f4db40f04a0 msgr2=0x7f4db40fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:08.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 shutdown_connections 2021-11-20T07:41:08.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:08.585+0000 7f4db8f53700 1 -- 172.21.15.71:0/7541728 wait complete. 2021-11-20T07:41:08.590 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:08.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:08.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:08 smithi112 conmon[36062]: cluster 2021-11-20T07:41:08.402465+0000 mgr.smithi071.ynqmwn (mgr.14162) 275 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:09.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:09 smithi071 conmon[33687]: audit 2021-11-20T07:41:08.585037+0000 mon.smithi071 (mon.0) 719 : audit [DBG] from='client.? 172.21.15.71:0/7541728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:09.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:09.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- 172.21.15.71:0/856504302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f55a0 msgr2=0x7fa8dc0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:09.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/856504302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc0f59c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa8cc004660 tx=0x7fa8cc009b30).stop 2021-11-20T07:41:09.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- 172.21.15.71:0/856504302 shutdown_connections 2021-11-20T07:41:09.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/856504302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/856504302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc0f59c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- 172.21.15.71:0/856504302 >> 172.21.15.71:0/856504302 conn(0x7fa8dc0f0ce0 msgr2=0x7fa8dc0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:09.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- 172.21.15.71:0/856504302 shutdown_connections 2021-11-20T07:41:09.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- 172.21.15.71:0/856504302 wait complete. 2021-11-20T07:41:09.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 Processor -- start 2021-11-20T07:41:09.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.727+0000 7fa8e2cab700 1 -- start start 2021-11-20T07:41:09.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:09.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:09.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc10c1b0 con 0x7fa8dc0f6840 2021-11-20T07:41:09.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc10c2f0 con 0x7fa8dc0f55a0 2021-11-20T07:41:09.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:09.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48760/0 (socket says 172.21.15.71:48760) 2021-11-20T07:41:09.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 -- 172.21.15.71:0/3260070931 learned_addr learned my addr 172.21.15.71:0/3260070931 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:09.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e0a47700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:09.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 -- 172.21.15.71:0/3260070931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f55a0 msgr2=0x7fa8dc05ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:09.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc05ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 -- 172.21.15.71:0/3260070931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc005040 con 0x7fa8dc0f6840 2021-11-20T07:41:09.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8dbfff700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa8d000c560 tx=0x7fa8d0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:09.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d000f650 con 0x7fa8dc0f6840 2021-11-20T07:41:09.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc10c550 con 0x7fa8dc0f6840 2021-11-20T07:41:09.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8d000fc60 con 0x7fa8dc0f6840 2021-11-20T07:41:09.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d0012c10 con 0x7fa8dc0f6840 2021-11-20T07:41:09.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.728+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc10cb10 con 0x7fa8dc0f6840 2021-11-20T07:41:09.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.729+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc027a00 con 0x7fa8dc0f6840 2021-11-20T07:41:09.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.730+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa8d0010070 con 0x7fa8dc0f6840 2021-11-20T07:41:09.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.730+0000 7fa8d97fa700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 0x7fa8c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:09.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.730+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fa8d0075ed0 con 0x7fa8dc0f6840 2021-11-20T07:41:09.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.730+0000 7fa8e0a47700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 0x7fa8c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:09.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.731+0000 7fa8e0a47700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 0x7fa8c405e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa8cc004630 tx=0x7fa8cc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:09.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.734+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8d00452b0 con 0x7fa8dc0f6840 2021-11-20T07:41:09.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:09 smithi112 conmon[36062]: audit 2021-11-20T07:41:08.585037+0000 mon.smithi071 (mon.0) 719 : audit [DBG] from='client.? 172.21.15.71:0/7541728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:09.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.917+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8dc0fdd30 con 0x7fa8dc0f6840 2021-11-20T07:41:09.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.918+0000 7fa8d97fa700 1 -- 172.21.15.71:0/3260070931 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7fa8d0048830 con 0x7fa8dc0f6840 2021-11-20T07:41:09.920 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:09.921 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:09.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 msgr2=0x7fa8c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:09.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 0x7fa8c405e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fa8cc004630 tx=0x7fa8cc009b30).stop 2021-11-20T07:41:09.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 msgr2=0x7fa8dc10bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:09.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa8d000c560 tx=0x7fa8d0007e20).stop 2021-11-20T07:41:09.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 shutdown_connections 2021-11-20T07:41:09.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8dc0f55a0 0x7fa8dc05ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa8c405c2a0 0x7fa8c405e760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 --2- 172.21.15.71:0/3260070931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8dc0f6840 0x7fa8dc10bcc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:09.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 >> 172.21.15.71:0/3260070931 conn(0x7fa8dc0f0ce0 msgr2=0x7fa8dc0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:09.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 shutdown_connections 2021-11-20T07:41:09.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:09.920+0000 7fa8e2cab700 1 -- 172.21.15.71:0/3260070931 wait complete. 2021-11-20T07:41:09.925 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:09.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:09.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:10 smithi112 conmon[36062]: audit 2021-11-20T07:41:09.919422+0000 mon.smithi071 (mon.0) 720 : audit [DBG] from='client.? 172.21.15.71:0/3260070931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:10 smithi112 conmon[36062]: cluster 2021-11-20T07:41:10.402787+0000 mgr.smithi071.ynqmwn (mgr.14162) 276 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 396 B/s wr, 1 op/s 2021-11-20T07:41:10.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:10 smithi071 conmon[33687]: audit 2021-11-20T07:41:09.919422+0000 mon.smithi071 (mon.0) 720 : audit 2021-11-20T07:41:10.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:10 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/3260070931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:10.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:10 smithi071 conmon[33687]: cluster 2021-11-20T07:41:10.402787+0000 mgr.smithi071.ynqmwn (mgr.14162) 276 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 396 B/s wr, 1 op/s 2021-11-20T07:41:10.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:11.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 -- 172.21.15.71:0/3545830006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 msgr2=0x7f9a580fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:11.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 --2- 172.21.15.71:0/3545830006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a580fb5a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9a48004660 tx=0x7f9a48009b30).stop 2021-11-20T07:41:11.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 -- 172.21.15.71:0/3545830006 shutdown_connections 2021-11-20T07:41:11.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 --2- 172.21.15.71:0/3545830006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 0x7f9a580fde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 --2- 172.21.15.71:0/3545830006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a580fb5a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 -- 172.21.15.71:0/3545830006 >> 172.21.15.71:0/3545830006 conn(0x7f9a580f0c00 msgr2=0x7f9a580f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:11.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 -- 172.21.15.71:0/3545830006 shutdown_connections 2021-11-20T07:41:11.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.062+0000 7f9a57fff700 1 -- 172.21.15.71:0/3545830006 wait complete. 2021-11-20T07:41:11.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 Processor -- start 2021-11-20T07:41:11.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 -- start start 2021-11-20T07:41:11.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:11.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 0x7f9a58109440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:11.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a581099f0 con 0x7f9a58060470 2021-11-20T07:41:11.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a57fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a58109b30 con 0x7f9a580fba90 2021-11-20T07:41:11.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a56ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:11.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a56ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48768/0 (socket says 172.21.15.71:48768) 2021-11-20T07:41:11.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a56ffd700 1 -- 172.21.15.71:0/1211741149 learned_addr learned my addr 172.21.15.71:0/1211741149 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:11.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a567fc700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 0x7f9a58109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:11.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a56ffd700 1 -- 172.21.15.71:0/1211741149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 msgr2=0x7f9a58109440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:11.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.063+0000 7f9a56ffd700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 0x7f9a58109440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a56ffd700 1 -- 172.21.15.71:0/1211741149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48005040 con 0x7f9a58060470 2021-11-20T07:41:11.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a56ffd700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9a48009f50 tx=0x7f9a48009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:11.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a48018070 con 0x7f9a58060470 2021-11-20T07:41:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5810a470 con 0x7f9a58060470 2021-11-20T07:41:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a48013410 con 0x7f9a58060470 2021-11-20T07:41:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a4801c3c0 con 0x7f9a58060470 2021-11-20T07:41:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.064+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5810a9d0 con 0x7f9a58060470 2021-11-20T07:41:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.065+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a58045ad0 con 0x7f9a58060470 2021-11-20T07:41:11.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.065+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9a4801d420 con 0x7f9a58060470 2021-11-20T07:41:11.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.065+0000 7f9a3f7fe700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 0x7f9a4005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:11.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.065+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9a48077fe0 con 0x7f9a58060470 2021-11-20T07:41:11.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.065+0000 7f9a567fc700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 0x7f9a4005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:11.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.066+0000 7f9a567fc700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 0x7f9a4005e7c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9a58060240 tx=0x7f9a4c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:11.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.069+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a4804a8c0 con 0x7f9a58060470 2021-11-20T07:41:11.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.253+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a58027a00 con 0x7f9a58060470 2021-11-20T07:41:11.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.253+0000 7f9a3f7fe700 1 -- 172.21.15.71:0/1211741149 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f9a4804dee0 con 0x7f9a58060470 2021-11-20T07:41:11.255 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:11.267 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:11.270 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 msgr2=0x7f9a4005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:11.270 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 0x7f9a4005e7c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9a58060240 tx=0x7f9a4c00b040).stop 2021-11-20T07:41:11.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 msgr2=0x7f9a58108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:11.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9a48009f50 tx=0x7f9a48009b30).stop 2021-11-20T07:41:11.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 shutdown_connections 2021-11-20T07:41:11.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9a4005c300 0x7f9a4005e7c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a58060470 0x7f9a58108f50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 --2- 172.21.15.71:0/1211741149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a580fba90 0x7f9a58109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 >> 172.21.15.71:0/1211741149 conn(0x7f9a580f0c00 msgr2=0x7f9a580f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 shutdown_connections 2021-11-20T07:41:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:11.255+0000 7f9a57fff700 1 -- 172.21.15.71:0/1211741149 wait complete. 2021-11-20T07:41:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:11.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:11.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:11.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:11 smithi071 conmon[33687]: audit 2021-11-20T07:41:11.254542+0000 mon.smithi071 (mon.0) 721 : audit [DBG] from='client.? 172.21.15.71:0/1211741149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:11 smithi112 conmon[36062]: audit 2021-11-20T07:41:11.254542+0000 mon.smithi071 (mon.0) 721 : audit [DBG] from='client.? 172.21.15.71:0/1211741149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:12.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:12.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 -- 172.21.15.71:0/830973582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 msgr2=0x7f9bdc0f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:12.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/830973582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc0f7090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9bd0004660 tx=0x7f9bd0009b30).stop 2021-11-20T07:41:12.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 -- 172.21.15.71:0/830973582 shutdown_connections 2021-11-20T07:41:12.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/830973582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 0x7f9bdc0f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:12.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/830973582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc0f7090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:12.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 -- 172.21.15.71:0/830973582 >> 172.21.15.71:0/830973582 conn(0x7f9bdc0f04a0 msgr2=0x7f9bdc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:12.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 -- 172.21.15.71:0/830973582 shutdown_connections 2021-11-20T07:41:12.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.399+0000 7f9be2fd6700 1 -- 172.21.15.71:0/830973582 wait complete. 2021-11-20T07:41:12.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 Processor -- start 2021-11-20T07:41:12.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 -- start start 2021-11-20T07:41:12.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 0x7f9bdc0f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:12.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:12.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc10c280 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be2fd6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc10c3c0 con 0x7f9bdc0f5970 2021-11-20T07:41:12.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:12.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48780/0 (socket says 172.21.15.71:48780) 2021-11-20T07:41:12.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 -- 172.21.15.71:0/3546483828 learned_addr learned my addr 172.21.15.71:0/3546483828 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:12.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9be0d72700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 0x7f9bdc0f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:12.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 -- 172.21.15.71:0/3546483828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 msgr2=0x7f9bdc0f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:12.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 0x7f9bdc0f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:12.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.400+0000 7f9bdbfff700 1 -- 172.21.15.71:0/3546483828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0005040 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9bdbfff700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9bc800b410 tx=0x7f9bc8006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:12.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc80125f0 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc10c6a0 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bc800be30 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc10cc60 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.401+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc8002a20 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.402+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc027a00 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.402+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9bc800f030 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.402+0000 7f9bd97fa700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 0x7f9bcc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:12.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.402+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9bc8076610 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.403+0000 7f9be0d72700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 0x7f9bcc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:12.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.403+0000 7f9be0d72700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 0x7f9bcc05e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9bd0004630 tx=0x7f9bd0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:12.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.406+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bc8048f70 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:12 smithi071 conmon[33687]: cluster 2021-11-20T07:41:12.403505+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:41:12.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:12 smithi071 conmon[33687]: 14162) 277 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:12.595 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.594+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bdc0fdea0 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.596 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.594+0000 7f9bd97fa700 1 -- 172.21.15.71:0/3546483828 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f9bc804c590 con 0x7f9bdc0f6c70 2021-11-20T07:41:12.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:12.597 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:12.598 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 msgr2=0x7f9bcc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:12.598 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 0x7f9bcc05e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9bd0004630 tx=0x7f9bd0009b30).stop 2021-11-20T07:41:12.598 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 msgr2=0x7f9bdc10bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:12.599 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9bc800b410 tx=0x7f9bc8006ef0).stop 2021-11-20T07:41:12.599 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 shutdown_connections 2021-11-20T07:41:12.599 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bdc0f5970 0x7f9bdc0f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:12.599 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9bcc05c2a0 0x7f9bcc05e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:12.600 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 --2- 172.21.15.71:0/3546483828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0f6c70 0x7f9bdc10bd90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:12.600 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 >> 172.21.15.71:0/3546483828 conn(0x7f9bdc0f04a0 msgr2=0x7f9bdc0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:12.600 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 shutdown_connections 2021-11-20T07:41:12.600 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:12.596+0000 7f9be2fd6700 1 -- 172.21.15.71:0/3546483828 wait complete. 2021-11-20T07:41:12.601 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:12.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:12.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:12 smithi112 conmon[36062]: cluster 2021-11-20T07:41:12.403505+0000 mgr.smithi071.ynqmwn (mgr.14162) 277 : cluster 2021-11-20T07:41:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:12 smithi112 conmon[36062]: [DBG] pgmap v248: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:13.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:13 smithi071 conmon[33687]: audit 2021-11-20T07:41:12.595423+0000 mon.smithi071 (mon.0) 722 : audit 2021-11-20T07:41:13.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:13 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/3546483828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:13.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:13.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.736+0000 7f971591c700 1 -- 172.21.15.71:0/1877214050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 msgr2=0x7f97100f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:13.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.736+0000 7f971591c700 1 --2- 172.21.15.71:0/1877214050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f5e70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9704004660 tx=0x7f9704009b30).stop 2021-11-20T07:41:13.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- 172.21.15.71:0/1877214050 shutdown_connections 2021-11-20T07:41:13.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 --2- 172.21.15.71:0/1877214050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97100f63f0 0x7f97100f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:13.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 --2- 172.21.15.71:0/1877214050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f5e70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- 172.21.15.71:0/1877214050 >> 172.21.15.71:0/1877214050 conn(0x7f97100f0360 msgr2=0x7f97100f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- 172.21.15.71:0/1877214050 shutdown_connections 2021-11-20T07:41:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- 172.21.15.71:0/1877214050 wait complete. 2021-11-20T07:41:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 Processor -- start 2021-11-20T07:41:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- start start 2021-11-20T07:41:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97100f63f0 0x7f97100f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100f8960 con 0x7f97100f5a50 2021-11-20T07:41:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.737+0000 7f971591c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100f8aa0 con 0x7f97100f63f0 2021-11-20T07:41:13.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:13.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48788/0 (socket says 172.21.15.71:48788) 2021-11-20T07:41:13.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 -- 172.21.15.71:0/1600333557 learned_addr learned my addr 172.21.15.71:0/1600333557 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:13.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 -- 172.21.15.71:0/1600333557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97100f63f0 msgr2=0x7f97100f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:13.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97100f63f0 0x7f97100f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:13.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 -- 172.21.15.71:0/1600333557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704005040 con 0x7f97100f5a50 2021-11-20T07:41:13.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971491a700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f9704004e30 tx=0x7f9704009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:13.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9704018070 con 0x7f97100f5a50 2021-11-20T07:41:13.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9704013410 con 0x7f97100f5a50 2021-11-20T07:41:13.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97100f8d20 con 0x7f97100f5a50 2021-11-20T07:41:13.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970401c3c0 con 0x7f97100f5a50 2021-11-20T07:41:13.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.738+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97100f9280 con 0x7f97100f5a50 2021-11-20T07:41:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.739+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9710045ad0 con 0x7f97100f5a50 2021-11-20T07:41:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.739+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f970401d420 con 0x7f97100f5a50 2021-11-20T07:41:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.740+0000 7f970d7fa700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 0x7f970005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.740+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9704077e20 con 0x7f97100f5a50 2021-11-20T07:41:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.740+0000 7f970ffff700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 0x7f970005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:13.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.740+0000 7f970ffff700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 0x7f970005e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f96fc004ad0 tx=0x7f96fc009af0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:13.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.743+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f970404a780 con 0x7f97100f5a50 2021-11-20T07:41:13.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:13 smithi112 conmon[36062]: audit 2021-11-20T07:41:12.595423+0000 mon.smithi071 (mon.0) 722 2021-11-20T07:41:13.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:13 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/3546483828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:13.929 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.926+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97100fdc10 con 0x7f97100f5a50 2021-11-20T07:41:13.929 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.926+0000 7f970d7fa700 1 -- 172.21.15.71:0/1600333557 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f970404dda0 con 0x7f97100f5a50 2021-11-20T07:41:13.931 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:13.931 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:13.934 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 msgr2=0x7f970005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:13.935 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 0x7f970005e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f96fc004ad0 tx=0x7f96fc009af0).stop 2021-11-20T07:41:13.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 msgr2=0x7f97100f9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:13.936 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f9704004e30 tx=0x7f9704009b30).stop 2021-11-20T07:41:13.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 shutdown_connections 2021-11-20T07:41:13.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970005c300 0x7f970005e7c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:13.937 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97100f5a50 0x7f97100f9f80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:13.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 --2- 172.21.15.71:0/1600333557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97100f63f0 0x7f97100f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:13.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 >> 172.21.15.71:0/1600333557 conn(0x7f97100f0360 msgr2=0x7f97100f4780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:13.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 shutdown_connections 2021-11-20T07:41:13.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:13.928+0000 7f971591c700 1 -- 172.21.15.71:0/1600333557 wait complete. 2021-11-20T07:41:13.940 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:13.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:13.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:14 smithi112 conmon[36062]: audit 2021-11-20T07:41:13.927715+0000 mon.smithi071 (mon.0) 723 : 2021-11-20T07:41:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:14 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/1600333557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:14.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:14 smithi112 conmon[36062]: cluster 2021-11-20T07:41:14.404112+0000 mgr.smithi071.ynqmwn (mgr.14162) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:14 smithi071 conmon[33687]: audit 2021-11-20T07:41:13.927715+0000 mon.smithi071 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.71:0/1600333557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:14 smithi071 conmon[33687]: cluster 2021-11-20T07:41 2021-11-20T07:41:14.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:14 smithi071 conmon[33687]: :14.404112+0000 mgr.smithi071.ynqmwn (mgr.14162) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:14.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:15.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 -- 172.21.15.71:0/4046695631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 msgr2=0x7f0ec40f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:15.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/4046695631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec40f6cb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f0eb8004660 tx=0x7f0eb8009b30).stop 2021-11-20T07:41:15.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 -- 172.21.15.71:0/4046695631 shutdown_connections 2021-11-20T07:41:15.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/4046695631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec40f6cb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/4046695631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 0x7f0ec40f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 -- 172.21.15.71:0/4046695631 >> 172.21.15.71:0/4046695631 conn(0x7f0ec40f0d10 msgr2=0x7f0ec40f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:15.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 -- 172.21.15.71:0/4046695631 shutdown_connections 2021-11-20T07:41:15.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 -- 172.21.15.71:0/4046695631 wait complete. 2021-11-20T07:41:15.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.067+0000 7f0ec9db5700 1 Processor -- start 2021-11-20T07:41:15.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec9db5700 1 -- start start 2021-11-20T07:41:15.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec9db5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 0x7f0ec4105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec9db5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec9db5700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41067c0 con 0x7f0ec40f6850 2021-11-20T07:41:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec9db5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec4106900 con 0x7f0ec40f55b0 2021-11-20T07:41:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:15.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48800/0 (socket says 172.21.15.71:48800) 2021-11-20T07:41:15.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 -- 172.21.15.71:0/3908790311 learned_addr learned my addr 172.21.15.71:0/3908790311 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:15.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec37fe700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 0x7f0ec4105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:15.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 -- 172.21.15.71:0/3908790311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 msgr2=0x7f0ec4105d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:15.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 0x7f0ec4105d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 -- 172.21.15.71:0/3908790311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb8005040 con 0x7f0ec40f6850 2021-11-20T07:41:15.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0ec2ffd700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f0eb8000c00 tx=0x7f0eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:15.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.068+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eb8019070 con 0x7f0ec40f6850 2021-11-20T07:41:15.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.069+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec410b340 con 0x7f0ec40f6850 2021-11-20T07:41:15.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.069+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0eb8007e70 con 0x7f0ec40f6850 2021-11-20T07:41:15.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.069+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eb8014c90 con 0x7f0ec40f6850 2021-11-20T07:41:15.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.069+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec410b900 con 0x7f0ec40f6850 2021-11-20T07:41:15.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.070+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec4046330 con 0x7f0ec40f6850 2021-11-20T07:41:15.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.070+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0eb8025070 con 0x7f0ec40f6850 2021-11-20T07:41:15.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.070+0000 7f0eabfff700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 0x7f0eac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:15.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.070+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f0eb8077280 con 0x7f0ec40f6850 2021-11-20T07:41:15.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.070+0000 7f0ec37fe700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 0x7f0eac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:15.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.071+0000 7f0ec37fe700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 0x7f0eac05e760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0eb4004fd0 tx=0x7f0eb400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:15.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.074+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0eb8049be0 con 0x7f0ec40f6850 2021-11-20T07:41:15.259 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.258+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec4027a00 con 0x7f0ec40f6850 2021-11-20T07:41:15.260 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.258+0000 7f0eabfff700 1 -- 172.21.15.71:0/3908790311 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f0ec4027a00 con 0x7f0ec40f6850 2021-11-20T07:41:15.260 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:15.261 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:15.262 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 msgr2=0x7f0eac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:15.262 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 0x7f0eac05e760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0eb4004fd0 tx=0x7f0eb400b040).stop 2021-11-20T07:41:15.262 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 msgr2=0x7f0ec4106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f0eb8000c00 tx=0x7f0eb800b040).stop 2021-11-20T07:41:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 shutdown_connections 2021-11-20T07:41:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ec40f55b0 0x7f0ec4105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0eac05c2a0 0x7f0eac05e760 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 --2- 172.21.15.71:0/3908790311 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40f6850 0x7f0ec4106210 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:15.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 >> 172.21.15.71:0/3908790311 conn(0x7f0ec40f0d10 msgr2=0x7f0ec40fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:15.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 shutdown_connections 2021-11-20T07:41:15.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:15.260+0000 7f0ec9db5700 1 -- 172.21.15.71:0/3908790311 wait complete. 2021-11-20T07:41:15.265 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:15.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:15.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:15.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:15 smithi071 conmon[33687]: audit 2021-11-20T07:41:15.259484+0000 mon.smithi071 (mon.0) 724 : audit [DBG] from='client.? 172.21.15.71:0/3908790311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:15.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:15 smithi112 conmon[36062]: audit 2021-11-20T07:41:15.259484+0000 mon.smithi071 ( 2021-11-20T07:41:15.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:15 smithi112 conmon[36062]: mon.0) 724 : audit [DBG] from='client.? 172.21.15.71:0/3908790311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:16.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:16.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.399+0000 7fd113fe7700 1 -- 172.21.15.71:0/2315389155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 msgr2=0x7fd10c0f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:16.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.399+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2315389155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c0f6a90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fd108004660 tx=0x7fd108009b30).stop 2021-11-20T07:41:16.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- 172.21.15.71:0/2315389155 shutdown_connections 2021-11-20T07:41:16.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2315389155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 0x7fd10c0f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2315389155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c0f6a90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- 172.21.15.71:0/2315389155 >> 172.21.15.71:0/2315389155 conn(0x7fd10c0f1db0 msgr2=0x7fd10c0f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- 172.21.15.71:0/2315389155 shutdown_connections 2021-11-20T07:41:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- 172.21.15.71:0/2315389155 wait complete. 2021-11-20T07:41:16.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 Processor -- start 2021-11-20T07:41:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- start start 2021-11-20T07:41:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:16.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 0x7fd10c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:16.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c107960 con 0x7fd10c0f6670 2021-11-20T07:41:16.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd113fe7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c107aa0 con 0x7fd10c0f7910 2021-11-20T07:41:16.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd111d83700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:16.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd111d83700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48808/0 (socket says 172.21.15.71:48808) 2021-11-20T07:41:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.400+0000 7fd111d83700 1 -- 172.21.15.71:0/2585236947 learned_addr learned my addr 172.21.15.71:0/2585236947 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd111582700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 0x7fd10c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd111d83700 1 -- 172.21.15.71:0/2585236947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 msgr2=0x7fd10c1073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:16.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd111d83700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 0x7fd10c1073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd111d83700 1 -- 172.21.15.71:0/2585236947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd108005040 con 0x7fd10c0f6670 2021-11-20T07:41:16.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd111d83700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd108007a70 tx=0x7fd108009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:16.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd108018070 con 0x7fd10c0f6670 2021-11-20T07:41:16.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c10c4e0 con 0x7fd10c0f6670 2021-11-20T07:41:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd108009700 con 0x7fd10c0f6670 2021-11-20T07:41:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd10801c620 con 0x7fd10c0f6670 2021-11-20T07:41:16.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.401+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c10caa0 con 0x7fd10c0f6670 2021-11-20T07:41:16.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.402+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c027a00 con 0x7fd10c0f6670 2021-11-20T07:41:16.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.402+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd10801d420 con 0x7fd10c0f6670 2021-11-20T07:41:16.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.403+0000 7fd1027fc700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 0x7fd0f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:16.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.403+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd108078110 con 0x7fd10c0f6670 2021-11-20T07:41:16.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.403+0000 7fd111582700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 0x7fd0f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:16.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.403+0000 7fd111582700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 0x7fd0f805e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd0fc007fd0 tx=0x7fd0fc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:16.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.407+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd10804aa70 con 0x7fd10c0f6670 2021-11-20T07:41:16.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:16 smithi071 conmon[33687]: cluster 2021-11-20T07:41:16.404489+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:41:16.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:16 smithi071 conmon[33687]: ) 279 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:41:16.600 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.598+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd10c0fee00 con 0x7fd10c0f6670 2021-11-20T07:41:16.601 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.599+0000 7fd1027fc700 1 -- 172.21.15.71:0/2585236947 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7fd10804e090 con 0x7fd10c0f6670 2021-11-20T07:41:16.602 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:16.602 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:16.607 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.600+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 msgr2=0x7fd0f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:16.607 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 0x7fd0f805e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd0fc007fd0 tx=0x7fd0fc009040).stop 2021-11-20T07:41:16.607 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 msgr2=0x7fd10c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:16.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd108007a70 tx=0x7fd108009b30).stop 2021-11-20T07:41:16.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 shutdown_connections 2021-11-20T07:41:16.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd0f805c2a0 0x7fd0f805e760 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.608 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd10c0f6670 0x7fd10c106ec0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 --2- 172.21.15.71:0/2585236947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd10c0f7910 0x7fd10c1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:16.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 >> 172.21.15.71:0/2585236947 conn(0x7fd10c0f1db0 msgr2=0x7fd10c0fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:16.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 shutdown_connections 2021-11-20T07:41:16.609 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:16.601+0000 7fd113fe7700 1 -- 172.21.15.71:0/2585236947 wait complete. 2021-11-20T07:41:16.609 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:16.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:16.614 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:16 smithi112 conmon[36062]: cluster 2021-11-20T07:41:16.404489+0000 mgr.smithi071.ynqmwn (mgr.14162) 279 : 2021-11-20T07:41:16.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:16 smithi112 conmon[36062]: cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:41:17.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:17.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 -- 172.21.15.71:0/4069583068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f54c0 msgr2=0x7f46f40f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 --2- 172.21.15.71:0/4069583068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f54c0 0x7f46f40f58e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f46e4004660 tx=0x7f46e4009b30).stop 2021-11-20T07:41:17.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 -- 172.21.15.71:0/4069583068 shutdown_connections 2021-11-20T07:41:17.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 --2- 172.21.15.71:0/4069583068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f6760 0x7f46f40f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 --2- 172.21.15.71:0/4069583068 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f54c0 0x7f46f40f58e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.740+0000 7f46f90a4700 1 -- 172.21.15.71:0/4069583068 >> 172.21.15.71:0/4069583068 conn(0x7f46f40f0c00 msgr2=0x7f46f40f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:17.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 -- 172.21.15.71:0/4069583068 shutdown_connections 2021-11-20T07:41:17.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 -- 172.21.15.71:0/4069583068 wait complete. 2021-11-20T07:41:17.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 Processor -- start 2021-11-20T07:41:17.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 -- start start 2021-11-20T07:41:17.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f54c0 0x7f46f4105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:17.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:17.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f41067d0 con 0x7f46f40f6760 2021-11-20T07:41:17.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.741+0000 7f46f90a4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f4106fd0 con 0x7f46f40f54c0 2021-11-20T07:41:17.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:17.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48820/0 (socket says 172.21.15.71:48820) 2021-11-20T07:41:17.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 -- 172.21.15.71:0/2637814185 learned_addr learned my addr 172.21.15.71:0/2637814185 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:17.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f3fff700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f54c0 0x7f46f4105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:17.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 -- 172.21.15.71:0/2637814185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f54c0 msgr2=0x7f46f4105d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:17.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f54c0 0x7f46f4105d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 -- 172.21.15.71:0/2637814185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4005040 con 0x7f46f40f6760 2021-11-20T07:41:17.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f37fe700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f46e80026d0 tx=0x7f46e8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:17.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46e80079c0 con 0x7f46f40f6760 2021-11-20T07:41:17.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f4107250 con 0x7f46f40f6760 2021-11-20T07:41:17.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46e800f040 con 0x7f46f40f6760 2021-11-20T07:41:17.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46e80148b0 con 0x7f46f40f6760 2021-11-20T07:41:17.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.742+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f41077c0 con 0x7f46f40f6760 2021-11-20T07:41:17.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.743+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f40f48b0 con 0x7f46f40f6760 2021-11-20T07:41:17.758 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.743+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f46e8007b20 con 0x7f46f40f6760 2021-11-20T07:41:17.758 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.744+0000 7f46f0ff9700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 0x7f46dc0609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:17.758 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.744+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f46e80761a0 con 0x7f46f40f6760 2021-11-20T07:41:17.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.744+0000 7f46f3fff700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 0x7f46dc0609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:17.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.744+0000 7f46f3fff700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 0x7f46dc0609d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f46e4009f50 tx=0x7f46e4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:17.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.748+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46e8048b00 con 0x7f46f40f6760 2021-11-20T07:41:17.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:17 smithi112 conmon[36062]: audit 2021-11-20T07:41:16.600390+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:17.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:17 smithi112 conmon[36062]: 725 : audit [DBG] from='client.? 172.21.15.71:0/2585236947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:17 smithi071 conmon[33687]: audit 2021-11-20T07:41:16.600390+0000 mon.smithi071 (mon.0) 725 : audit [DBG] from='client.? 172.21.15.71:0/2585236947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:17 smithi071 conmon[33687]: 2021-11-20T07:41:17.934 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.933+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46f40fdc90 con 0x7f46f40f6760 2021-11-20T07:41:17.935 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.933+0000 7f46f0ff9700 1 -- 172.21.15.71:0/2637814185 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f46e804c120 con 0x7f46f40f6760 2021-11-20T07:41:17.936 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:17.936 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:17.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 msgr2=0x7f46dc0609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:17.938 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 0x7f46dc0609d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f46e4009f50 tx=0x7f46e4009b30).stop 2021-11-20T07:41:17.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 msgr2=0x7f46f4106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:17.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f46e80026d0 tx=0x7f46e8009de0).stop 2021-11-20T07:41:17.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 shutdown_connections 2021-11-20T07:41:17.939 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f40f54c0 0x7f46f4105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46dc05e510 0x7f46dc0609d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 --2- 172.21.15.71:0/2637814185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46f40f6760 0x7f46f4106220 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:17.940 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 >> 172.21.15.71:0/2637814185 conn(0x7f46f40f0c00 msgr2=0x7f46f40fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:17.941 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 shutdown_connections 2021-11-20T07:41:17.941 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:17.935+0000 7f46f90a4700 1 -- 172.21.15.71:0/2637814185 wait complete. 2021-11-20T07:41:17.942 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:17.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:17.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:18.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:18 smithi112 conmon[36062]: audit 2021-11-20T07:41:17.934666+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:18 smithi112 conmon[36062]: 726 : audit [DBG] from='client.? 172.21.15.71:0/2637814185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:18 smithi112 conmon[36062]: cluster 2021-11-20T07:41:18.405155+0000 mgr.smithi071.ynqmwn (mgr.14162) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 85 B/s wr, 0 op/s 2021-11-20T07:41:18.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:18 smithi071 conmon[33687]: audit 2021-11-20T07:41:17.934666+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:18.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:18 smithi071 conmon[33687]: 726 : audit [DBG] from='client.? 172.21.15.71:0/2637814185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:18.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:18 smithi071 conmon[33687]: cluster 2021-11-20T07:41:18.405155+0000 mgr.smithi071.ynqmwn (mgr.14162) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 85 B/s wr, 0 op/s 2021-11-20T07:41:18.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 -- 172.21.15.71:0/3860224846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f6680 msgr2=0x7f293c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 --2- 172.21.15.71:0/3860224846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f6680 0x7f293c0f6aa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f292c0099e0 tx=0x7f292c004d10).stop 2021-11-20T07:41:19.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 -- 172.21.15.71:0/3860224846 shutdown_connections 2021-11-20T07:41:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 --2- 172.21.15.71:0/3860224846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f7920 0x7f293c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 --2- 172.21.15.71:0/3860224846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f6680 0x7f293c0f6aa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 -- 172.21.15.71:0/3860224846 >> 172.21.15.71:0/3860224846 conn(0x7f293c0f1de0 msgr2=0x7f293c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 -- 172.21.15.71:0/3860224846 shutdown_connections 2021-11-20T07:41:19.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.075+0000 7f2944410700 1 -- 172.21.15.71:0/3860224846 wait complete. 2021-11-20T07:41:19.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 Processor -- start 2021-11-20T07:41:19.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 -- start start 2021-11-20T07:41:19.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f6680 0x7f293c107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:19.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:19.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c107b80 con 0x7f293c0f7920 2021-11-20T07:41:19.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c107cc0 con 0x7f293c0f6680 2021-11-20T07:41:19.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:19.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48830/0 (socket says 172.21.15.71:48830) 2021-11-20T07:41:19.083 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 -- 172.21.15.71:0/1007085167 learned_addr learned my addr 172.21.15.71:0/1007085167 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:19.083 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29421ac700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f6680 0x7f293c107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:19.083 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 -- 172.21.15.71:0/1007085167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f6680 msgr2=0x7f293c107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:19.083 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f6680 0x7f293c107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:19.083 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 -- 172.21.15.71:0/1007085167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c009640 con 0x7f293c0f7920 2021-11-20T07:41:19.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f29419ab700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f293000b410 tx=0x7f2930006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:19.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29300125f0 con 0x7f293c0f7920 2021-11-20T07:41:19.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f293000be80 con 0x7f293c0f7920 2021-11-20T07:41:19.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c10c760 con 0x7f293c0f7920 2021-11-20T07:41:19.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.076+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2930002ac0 con 0x7f293c0f7920 2021-11-20T07:41:19.085 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.077+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c10cca0 con 0x7f293c0f7920 2021-11-20T07:41:19.085 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.077+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c027a00 con 0x7f293c0f7920 2021-11-20T07:41:19.086 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.078+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f293000f030 con 0x7f293c0f7920 2021-11-20T07:41:19.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.078+0000 7f293affd700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 0x7f292405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:19.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.078+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f2930076700 con 0x7f293c0f7920 2021-11-20T07:41:19.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.078+0000 7f29421ac700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 0x7f292405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:19.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.078+0000 7f29421ac700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 0x7f292405e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f292c00b3b0 tx=0x7f292c004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:19.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.082+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2930049000 con 0x7f293c0f7920 2021-11-20T07:41:19.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.266+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f293c0fee90 con 0x7f293c0f7920 2021-11-20T07:41:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.266+0000 7f293affd700 1 -- 172.21.15.71:0/1007085167 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f293004c620 con 0x7f293c0f7920 2021-11-20T07:41:19.269 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:19.269 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:04.409745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24369,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24369":{"gid":24369,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":51,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.112:6827/3888903196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3888903196},{"type":"v1","addr":"172.21.15.112:6827","nonce":3888903196}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:19.270 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.267+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 msgr2=0x7f292405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:19.270 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.267+0000 7f2944410700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 0x7f292405e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f292c00b3b0 tx=0x7f292c004d10).stop 2021-11-20T07:41:19.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.267+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 msgr2=0x7f293c107540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:19.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f293000b410 tx=0x7f2930006ef0).stop 2021-11-20T07:41:19.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 shutdown_connections 2021-11-20T07:41:19.271 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f293c0f6680 0x7f293c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:19.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f292405c2a0 0x7f292405e760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:19.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 --2- 172.21.15.71:0/1007085167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f293c0f7920 0x7f293c107540 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:19.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 >> 172.21.15.71:0/1007085167 conn(0x7f293c0f1de0 msgr2=0x7f293c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:19.272 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 shutdown_connections 2021-11-20T07:41:19.273 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:19.268+0000 7f2944410700 1 -- 172.21.15.71:0/1007085167 wait complete. 2021-11-20T07:41:19.273 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 58 2021-11-20T07:41:19.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:04.409745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24369, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24369': {'gid': 24369, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 51, 'state': 'up:clientreplay', 'state_seq': 22, 'addr': '172.21.15.112:6827/3888903196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 3888903196}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 3888903196}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:19.282 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24369:up:clientreplay 2021-11-20T07:41:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: audit 2021-11-20T07:41:18.938133+0000 mon.smithi071 (mon.0) 727 2021-11-20T07:41:19.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:41:19.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: audit 2021-11-20T07:41:18.938583+0000 mon.smithi071 (mon.0) 728 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:41:19.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: audit 2021-11-20T07: 2021-11-20T07:41:19.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: 41:19.267457+0000 mon.smithi071 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.71:0/1007085167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:19.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: cluster 2021-11-20T07:41: 2021-11-20T07:41:19.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: 19.287355+0000 mon.smithi071 (mon.0) 730 : cluster [INF] daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:41:19.571 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: cluster 2021-11-20T07:41:19.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: 2021-11-20T07:41: 2021-11-20T07:41:19.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: 19.291099+0000 mon.smithi071 (mon.0) 731 : 2021-11-20T07:41:19.572 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:19 smithi071 conmon[33687]: cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T07:41:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: audit 2021-11-20T07:41:18.938133+0000 mon.smithi071 (mon.0) 727 : audit [INF] 2021-11-20T07:41:19.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:41:19.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: audit 2021-11-20T07:41:18.938583+0000 mon.smithi071 (mon.0) 728 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:41:19.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: audit 2021-11-20T07:41 2021-11-20T07:41:19.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: :19.267457+0000 mon.smithi071 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.71:0/1007085167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:19.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:41:19.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: -11-20T07:41:19.287355+0000 mon.smithi071 (mon.0) 730 : cluster [INF] 2021-11-20T07:41:19.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:41:19.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: cluster 2021-11-20T07:41: 2021-11-20T07:41:19.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: 19.291099+0000 mon.smithi071 (mon.0) 731 : cluster [DBG] 2021-11-20T07:41:19.788 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:19 smithi112 conmon[36062]: osdmap e52: 6 total, 6 up, 6 in 2021-11-20T07:41:20.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:20.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- 172.21.15.71:0/420392337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f6840 msgr2=0x7f31500f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:20.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 --2- 172.21.15.71:0/420392337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f6840 0x7f31500f6ca0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f313c004660 tx=0x7f313c009b30).stop 2021-11-20T07:41:20.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- 172.21.15.71:0/420392337 shutdown_connections 2021-11-20T07:41:20.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 --2- 172.21.15.71:0/420392337 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f6840 0x7f31500f6ca0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 --2- 172.21.15.71:0/420392337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f55a0 0x7f31500f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- 172.21.15.71:0/420392337 >> 172.21.15.71:0/420392337 conn(0x7f31500f0ce0 msgr2=0x7f31500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:20.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- 172.21.15.71:0/420392337 shutdown_connections 2021-11-20T07:41:20.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- 172.21.15.71:0/420392337 wait complete. 2021-11-20T07:41:20.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 Processor -- start 2021-11-20T07:41:20.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.402+0000 7f3157225700 1 -- start start 2021-11-20T07:41:20.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:20.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f6840 0x7f31500f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:20.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31500fa2f0 con 0x7f31500f55a0 2021-11-20T07:41:20.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31500f8890 con 0x7f31500f6840 2021-11-20T07:41:20.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:20.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48850/0 (socket says 172.21.15.71:48850) 2021-11-20T07:41:20.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 -- 172.21.15.71:0/3618645846 learned_addr learned my addr 172.21.15.71:0/3618645846 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:20.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f314ffff700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f6840 0x7f31500f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 -- 172.21.15.71:0/3618645846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f6840 msgr2=0x7f31500f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f6840 0x7f31500f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 -- 172.21.15.71:0/3618645846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c005040 con 0x7f31500f55a0 2021-11-20T07:41:20.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3154fc1700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3144002700 tx=0x7f3144009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31440079c0 con 0x7f31500f55a0 2021-11-20T07:41:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31500f8b10 con 0x7f31500f55a0 2021-11-20T07:41:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f314400f040 con 0x7f31500f55a0 2021-11-20T07:41:20.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3144014860 con 0x7f31500f55a0 2021-11-20T07:41:20.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.403+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31500f90d0 con 0x7f31500f55a0 2021-11-20T07:41:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.404+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31500fdd30 con 0x7f31500f55a0 2021-11-20T07:41:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.405+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f314401b060 con 0x7f31500f55a0 2021-11-20T07:41:20.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.405+0000 7f314d7fa700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 0x7f314005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.405+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f31440760c0 con 0x7f31500f55a0 2021-11-20T07:41:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.405+0000 7f314ffff700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 0x7f314005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.406+0000 7f314ffff700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 0x7f314005e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f313c009f50 tx=0x7f313c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.410+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31440489d0 con 0x7f31500f55a0 2021-11-20T07:41:20.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.611+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f315010f4a0 con 0x7f31500f55a0 2021-11-20T07:41:20.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.611+0000 7f314d7fa700 1 -- 172.21.15.71:0/3618645846 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2947 (secure 0 0 0) 0x7f314404bff0 con 0x7f31500f55a0 2021-11-20T07:41:20.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:20.614 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:20.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 msgr2=0x7f314005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:20.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 0x7f314005e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f313c009f50 tx=0x7f313c009b30).stop 2021-11-20T07:41:20.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 msgr2=0x7f31500f9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:20.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3144002700 tx=0x7f3144009d80).stop 2021-11-20T07:41:20.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 shutdown_connections 2021-11-20T07:41:20.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f314005c2a0 0x7f314005e760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31500f55a0 0x7f31500f9d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 --2- 172.21.15.71:0/3618645846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31500f6840 0x7f31500f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:20.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 >> 172.21.15.71:0/3618645846 conn(0x7f31500f0ce0 msgr2=0x7f315005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:20.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 shutdown_connections 2021-11-20T07:41:20.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:20.613+0000 7f3157225700 1 -- 172.21.15.71:0/3618645846 wait complete. 2021-11-20T07:41:20.618 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 61 2021-11-20T07:41:20.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:20.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:20.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: cluster 2021-11-20T07:41:19.466731+0000 mon.smithi071 (mon.0) 732 : cluster [DBG] 2021-11-20T07:41:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:boot 2021-11-20T07:41:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: cluster 2021-11-20T07:41:19.466775+0000 mon.smithi071 (mon.0) 733 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 2021-11-20T07:41:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: cluster 2021-11-20T07:41:19.467035+0000 mon.smithi071 (mon.0) 734 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.zaejme=up:active} 3 up:standby, 1 failed 2021-11-20T07:41:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: audit 2021-11-20T07:41:19.467130+0000 mon.smithi071 (mon.0) 735 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:41:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: cluster 2021-11-20T07:41:19.469585+0000 mon.smithi071 (mon.0) 736 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:replay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:20.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:20 smithi112 conmon[36062]: cluster 2021-11-20T07:41:20.405558+0000 mgr.smithi071.ynqmwn (mgr.14162) 281 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:41:20.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: cluster 2021-11-20T07:41:19.466731+0000 mon.smithi071 (mon.0 2021-11-20T07:41:20.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: ) 732 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:boot 2021-11-20T07:41:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: cluster 2021-11-20T07:41:19.466775+0000 mon.smithi071 (mon.0) 733 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 0 2021-11-20T07:41:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: cluster 2021-11-20T07:41:19.467035+0000 mon.smithi071 (mon.0) 734 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi071.zaejme=up:active} 3 up:standby, 1 failed 2021-11-20T07:41:20.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: audit 2021-11-20T07:41:19.467130+0000 mon.smithi071 (mon.0) 735 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:41:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: cluster 2021-11-20T07:41:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: 2021-11-20T07:41:19.469585+0000 mon.smithi071 (mon.0) 736 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:replay,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:41:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: 07:41:20.405558+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:41:20.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:20 smithi071 conmon[33687]: ) 281 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:41:21.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 -- 172.21.15.71:0/4119586152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f7550 msgr2=0x7f26fc0f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 --2- 172.21.15.71:0/4119586152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f7550 0x7f26fc0f5970 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f26ec004660 tx=0x7f26ec009b30).stop 2021-11-20T07:41:21.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 -- 172.21.15.71:0/4119586152 shutdown_connections 2021-11-20T07:41:21.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 --2- 172.21.15.71:0/4119586152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f7550 0x7f26fc0f5970 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 --2- 172.21.15.71:0/4119586152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 -- 172.21.15.71:0/4119586152 >> 172.21.15.71:0/4119586152 conn(0x7f26fc0f0440 msgr2=0x7f26fc0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 -- 172.21.15.71:0/4119586152 shutdown_connections 2021-11-20T07:41:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.755+0000 7f270467c700 1 -- 172.21.15.71:0/4119586152 wait complete. 2021-11-20T07:41:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 Processor -- start 2021-11-20T07:41:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 -- start start 2021-11-20T07:41:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f7550 0x7f26fc10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc10c100 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f270467c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc10c240 con 0x7f26fc0f7550 2021-11-20T07:41:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48870/0 (socket says 172.21.15.71:48870) 2021-11-20T07:41:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 -- 172.21.15.71:0/817114495 learned_addr learned my addr 172.21.15.71:0/817114495 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2701c17700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f7550 0x7f26fc10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 -- 172.21.15.71:0/817114495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f7550 msgr2=0x7f26fc10bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:21.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f7550 0x7f26fc10bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 -- 172.21.15.71:0/817114495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec005040 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.756+0000 7f2702418700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f26f800c530 tx=0x7f26f8007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:21.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.757+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f800f650 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.757+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26f800fc60 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.757+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc10c4d0 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.757+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f8012d20 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.757+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc10ca90 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.758+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc027a00 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.758+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f26f8019030 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.758+0000 7f26f2ffd700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 0x7f26e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.758+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f26f8075f20 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.758+0000 7f2701c17700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 0x7f26e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:21.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.759+0000 7f2701c17700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 0x7f26e805e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f26ec013580 tx=0x7f26ec009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:21.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.762+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26f80a22b0 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:21 smithi112 conmon[36062]: cluster 2021-11-20T07:41:20.471837+0000 mon.smithi071 (mon.0) 737 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:resolve 2021-11-20T07:41:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:21 smithi112 conmon[36062]: cluster 2021-11-20T07:41:20.471895 2021-11-20T07:41:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:21 smithi112 conmon[36062]: +0000 mon.smithi071 (mon.0) 738 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:21.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:21 smithi112 conmon[36062]: audit 2021-11-20T07:41:20.612506+0000 mon.smithi071 (mon.0) 739 : audit [DBG] from='client.? 172.21.15.71:0/3618645846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:21 smithi071 conmon[33687]: cluster 2021-11-20T07:41:20.471837+0000 mon.smithi071 (mon.0) 737 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:resolve 2021-11-20T07:41:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:21 smithi071 conmon[33687]: cluster 2021-11-20T07:41: 2021-11-20T07:41:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:21 smithi071 conmon[33687]: 20.471895+0000 mon.smithi071 (mon.0) 738 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:21 smithi071 conmon[33687]: audit 2021-11-20T07:41:20.612506+0000 mon.smithi071 (mon.0) 739 : audit [DBG] from='client.? 172.21.15.71:0/3618645846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:21.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.945+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26fc0fdd40 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.945+0000 7f26f2ffd700 1 -- 172.21.15.71:0/817114495 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2947 (secure 0 0 0) 0x7f26f8017070 con 0x7f26fc0f6bb0 2021-11-20T07:41:21.948 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:21.948 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:21.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 msgr2=0x7f26e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:21.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 0x7f26e805e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f26ec013580 tx=0x7f26ec009b30).stop 2021-11-20T07:41:21.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 msgr2=0x7f26fc0f5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:21.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f26f800c530 tx=0x7f26f8007e10).stop 2021-11-20T07:41:21.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 shutdown_connections 2021-11-20T07:41:21.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f26e805c2a0 0x7f26e805e760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f26fc0f6bb0 0x7f26fc0f5350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 --2- 172.21.15.71:0/817114495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f26fc0f7550 0x7f26fc10bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:21.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 >> 172.21.15.71:0/817114495 conn(0x7f26fc0f0440 msgr2=0x7f26fc0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:21.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 shutdown_connections 2021-11-20T07:41:21.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:21.947+0000 7f270467c700 1 -- 172.21.15.71:0/817114495 wait complete. 2021-11-20T07:41:21.953 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 61 2021-11-20T07:41:21.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:21.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:22.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:22 smithi112 conmon[36062]: audit 2021-11-20T07:41:21.946477+0000 mon.smithi071 (mon.0) 740 : audit [DBG] from='client.? 172.21.15.71:0/817114495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:22.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:22 smithi112 conmon[36062]: cluster 2021-11-20T07:41:22.406133+0000 mgr.smithi071.ynqmwn (mgr.14162) 282 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:41:22.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:22 smithi071 conmon[33687]: audit 2021-11-20T07:41:21.946477+0000 mon.smithi071 2021-11-20T07:41:22.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:22 smithi071 conmon[33687]: (mon.0) 740 : audit [DBG] from='client.? 172.21.15.71:0/817114495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:22.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:22 smithi071 conmon[33687]: cluster 2021-11-20T07:41:22.406133+0000 mgr.smithi071.ynqmwn (mgr.14162) 282 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:41:22.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:23.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- 172.21.15.71:0/3193728855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 msgr2=0x7feaa80f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/3193728855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa80f5f20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fea980099e0 tx=0x7fea98004d10).stop 2021-11-20T07:41:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- 172.21.15.71:0/3193728855 shutdown_connections 2021-11-20T07:41:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/3193728855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feaa80f4780 0x7feaa80f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:23.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/3193728855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa80f5f20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- 172.21.15.71:0/3193728855 >> 172.21.15.71:0/3193728855 conn(0x7feaa80f04a0 msgr2=0x7feaa80f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- 172.21.15.71:0/3193728855 shutdown_connections 2021-11-20T07:41:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- 172.21.15.71:0/3193728855 wait complete. 2021-11-20T07:41:23.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 Processor -- start 2021-11-20T07:41:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.089+0000 7feaaf6a9700 1 -- start start 2021-11-20T07:41:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feaa80f4780 0x7feaa810d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa810e200 con 0x7feaa80f5b00 2021-11-20T07:41:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa810e340 con 0x7feaa80f4780 2021-11-20T07:41:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48886/0 (socket says 172.21.15.71:48886) 2021-11-20T07:41:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 -- 172.21.15.71:0/2044871778 learned_addr learned my addr 172.21.15.71:0/2044871778 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 -- 172.21.15.71:0/2044871778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feaa80f4780 msgr2=0x7feaa810d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feaa80f4780 0x7feaa810d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 -- 172.21.15.71:0/2044871778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98009640 con 0x7feaa80f5b00 2021-11-20T07:41:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaacc44700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fea9c00b410 tx=0x7fea9c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea9c0125f0 con 0x7feaa80f5b00 2021-11-20T07:41:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa8112de0 con 0x7feaa80f5b00 2021-11-20T07:41:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea9c00be80 con 0x7feaa80f5b00 2021-11-20T07:41:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea9c002ac0 con 0x7feaa80f5b00 2021-11-20T07:41:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.090+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa81132f0 con 0x7feaa80f5b00 2021-11-20T07:41:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.091+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa8046330 con 0x7feaa80f5b00 2021-11-20T07:41:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.092+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fea9c00f030 con 0x7feaa80f5b00 2021-11-20T07:41:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.092+0000 7feaa5ffb700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 0x7fea9005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.092+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fea9c076730 con 0x7feaa80f5b00 2021-11-20T07:41:23.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.092+0000 7feaad445700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 0x7fea9005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:23.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.092+0000 7feaad445700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 0x7fea9005e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fea98003380 tx=0x7fea98004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:23.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.096+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea9c048fe0 con 0x7feaa80f5b00 2021-11-20T07:41:23.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.279+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaa8027a00 con 0x7feaa80f5b00 2021-11-20T07:41:23.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.279+0000 7feaa5ffb700 1 -- 172.21.15.71:0/2044871778 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2947 (secure 0 0 0) 0x7fea9c04c600 con 0x7feaa80f5b00 2021-11-20T07:41:23.282 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:23.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:23.283 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 msgr2=0x7fea9005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:23.283 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 0x7fea9005e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fea98003380 tx=0x7fea98004d10).stop 2021-11-20T07:41:23.284 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 msgr2=0x7feaa810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:23.284 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fea9c00b410 tx=0x7fea9c006ef0).stop 2021-11-20T07:41:23.284 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 shutdown_connections 2021-11-20T07:41:23.284 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feaa80f4780 0x7feaa810d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:23.285 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea9005c2a0 0x7fea9005e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:23.285 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 --2- 172.21.15.71:0/2044871778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feaa80f5b00 0x7feaa810dbc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:23.285 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 >> 172.21.15.71:0/2044871778 conn(0x7feaa80f04a0 msgr2=0x7feaa80f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:23.285 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 shutdown_connections 2021-11-20T07:41:23.286 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:23.281+0000 7feaaf6a9700 1 -- 172.21.15.71:0/2044871778 wait complete. 2021-11-20T07:41:23.286 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 61 2021-11-20T07:41:23.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:23.295 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:23.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:23 smithi071 conmon[33687]: audit 2021-11-20T07:41:23.280776+0000 mon.smithi071 2021-11-20T07:41:23.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:23 smithi071 conmon[33687]: (mon.0) 741 : audit [DBG] from='client.? 172.21.15.71:0/2044871778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:23.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:23 smithi112 conmon[36062]: audit 2021-11-20T07:41:23.280776+0000 mon.smithi071 (mon.0) 741 : audit [DBG] from='client.? 172.21.15.71:0/2044871778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:24.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:24.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 -- 172.21.15.71:0/139884150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 msgr2=0x7f46cc0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:24.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 --2- 172.21.15.71:0/139884150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0f5990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f46c8004660 tx=0x7f46c8009b30).stop 2021-11-20T07:41:24.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 -- 172.21.15.71:0/139884150 shutdown_connections 2021-11-20T07:41:24.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 --2- 172.21.15.71:0/139884150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 0x7f46cc0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:24.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 --2- 172.21.15.71:0/139884150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0f5990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:24.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 -- 172.21.15.71:0/139884150 >> 172.21.15.71:0/139884150 conn(0x7f46cc0f0cb0 msgr2=0x7f46cc0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:24.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 -- 172.21.15.71:0/139884150 shutdown_connections 2021-11-20T07:41:24.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.414+0000 7f46d41d0700 1 -- 172.21.15.71:0/139884150 wait complete. 2021-11-20T07:41:24.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 Processor -- start 2021-11-20T07:41:24.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 -- start start 2021-11-20T07:41:24.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:24.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 0x7f46cc10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10c1a0 con 0x7f46cc0f5570 2021-11-20T07:41:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d41d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10c2e0 con 0x7f46cc0f6810 2021-11-20T07:41:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48898/0 (socket says 172.21.15.71:48898) 2021-11-20T07:41:24.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 -- 172.21.15.71:0/3071503899 learned_addr learned my addr 172.21.15.71:0/3071503899 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:24.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d176b700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 0x7f46cc10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:24.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 -- 172.21.15.71:0/3071503899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 msgr2=0x7f46cc10bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:24.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 0x7f46cc10bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:24.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 -- 172.21.15.71:0/3071503899 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c8005040 con 0x7f46cc0f5570 2021-11-20T07:41:24.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.415+0000 7f46d1f6c700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f46c8009f50 tx=0x7f46c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:24.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.416+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c8018070 con 0x7f46cc0f5570 2021-11-20T07:41:24.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.416+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46c8013410 con 0x7f46cc0f5570 2021-11-20T07:41:24.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.416+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c801c620 con 0x7f46cc0f5570 2021-11-20T07:41:24.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.416+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc10c560 con 0x7f46cc0f5570 2021-11-20T07:41:24.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.416+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc10cac0 con 0x7f46cc0f5570 2021-11-20T07:41:24.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.417+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f46c801d420 con 0x7f46cc0f5570 2021-11-20T07:41:24.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.417+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc027a00 con 0x7f46cc0f5570 2021-11-20T07:41:24.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.417+0000 7f46c27fc700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 0x7f46b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:24.429 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.417+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f46c8078040 con 0x7f46cc0f5570 2021-11-20T07:41:24.429 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.418+0000 7f46d176b700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 0x7f46b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:24.429 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.418+0000 7f46d176b700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 0x7f46b805e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f46bc00b790 tx=0x7f46bc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:24.430 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.422+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46c804a9d0 con 0x7f46cc0f5570 2021-11-20T07:41:24.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:24 smithi071 conmon[33687]: cluster 2021-11-20T07:41:23.483684 2021-11-20T07:41:24.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:24 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 742 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:standby 2021-11-20T07:41:24.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:24 smithi071 conmon[33687]: cluster 2021-11-20T07:41:23.483735+0000 mon.smithi071 (mon.0) 743 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:24.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:24 smithi071 conmon[33687]: cluster 2021-11-20T07:41:24.406518+0000 mgr.smithi071.ynqmwn (mgr.14162) 283 2021-11-20T07:41:24.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:24 smithi071 conmon[33687]: : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:24.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.610+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46cc0fdd80 con 0x7f46cc0f5570 2021-11-20T07:41:24.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.611+0000 7f46c27fc700 1 -- 172.21.15.71:0/3071503899 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f46c804dff0 con 0x7f46cc0f5570 2021-11-20T07:41:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:24.615 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:24.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 msgr2=0x7f46b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:24.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 0x7f46b805e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f46bc00b790 tx=0x7f46bc008040).stop 2021-11-20T07:41:24.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 msgr2=0x7f46cc0604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:24.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f46c8009f50 tx=0x7f46c8009b30).stop 2021-11-20T07:41:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 shutdown_connections 2021-11-20T07:41:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f46b805c2a0 0x7f46b805e760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46cc0f5570 0x7f46cc0604d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:24.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 --2- 172.21.15.71:0/3071503899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc0f6810 0x7f46cc10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:24.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 >> 172.21.15.71:0/3071503899 conn(0x7f46cc0f0cb0 msgr2=0x7f46cc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:24.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 shutdown_connections 2021-11-20T07:41:24.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:24.613+0000 7f46d41d0700 1 -- 172.21.15.71:0/3071503899 wait complete. 2021-11-20T07:41:24.618 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:24.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:24.626 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:24.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: cluster 2021-11-20T07:41:23.483684+0000 mon.smithi071 (mon.0) 742 : cluster 2021-11-20T07:41:24.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:standby 2021-11-20T07:41:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: cluster 2021-11-20T07:41:23.483735+0000 mon.smithi071 (mon.0) 743 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi071.zaejme=up:active} 2 up:standby 2021-11-20T07:41:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:41:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: 20T07:41:24.406518+0000 2021-11-20T07:41:24.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 283 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:24.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:24 smithi112 conmon[36062]: 2021-11-20T07:41:25.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:25.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 -- 172.21.15.71:0/1664605957 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f7940 msgr2=0x7f107c0f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:25.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 --2- 172.21.15.71:0/1664605957 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f7940 0x7f107c0f7da0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f1064004660 tx=0x7f1064009b30).stop 2021-11-20T07:41:25.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 -- 172.21.15.71:0/1664605957 shutdown_connections 2021-11-20T07:41:25.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 --2- 172.21.15.71:0/1664605957 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f7940 0x7f107c0f7da0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 --2- 172.21.15.71:0/1664605957 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f66a0 0x7f107c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 -- 172.21.15.71:0/1664605957 >> 172.21.15.71:0/1664605957 conn(0x7f107c0f1e00 msgr2=0x7f107c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:25.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 -- 172.21.15.71:0/1664605957 shutdown_connections 2021-11-20T07:41:25.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.749+0000 7f1081f89700 1 -- 172.21.15.71:0/1664605957 wait complete. 2021-11-20T07:41:25.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 Processor -- start 2021-11-20T07:41:25.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 -- start start 2021-11-20T07:41:25.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:25.758 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f7940 0x7f107c1072e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:25.758 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c107890 con 0x7f107c0f66a0 2021-11-20T07:41:25.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f1081f89700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1079d0 con 0x7f107c0f7940 2021-11-20T07:41:25.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:25.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48906/0 (socket says 172.21.15.71:48906) 2021-11-20T07:41:25.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 -- 172.21.15.71:0/3578370751 learned_addr learned my addr 172.21.15.71:0/3578370751 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:25.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107affd700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f7940 0x7f107c1072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:25.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 -- 172.21.15.71:0/3578370751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f7940 msgr2=0x7f107c1072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:25.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f7940 0x7f107c1072e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 -- 172.21.15.71:0/3578370751 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064005040 con 0x7f107c0f66a0 2021-11-20T07:41:25.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.750+0000 7f107b7fe700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f106c00e7c0 tx=0x7f106c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:25.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.751+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f106c0092e0 con 0x7f107c0f66a0 2021-11-20T07:41:25.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.751+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c10c470 con 0x7f107c0f66a0 2021-11-20T07:41:25.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.751+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f106c00f040 con 0x7f107c0f66a0 2021-11-20T07:41:25.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.751+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f106c0147e0 con 0x7f107c0f66a0 2021-11-20T07:41:25.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.751+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c10ca30 con 0x7f107c0f66a0 2021-11-20T07:41:25.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.752+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c027a00 con 0x7f107c0f66a0 2021-11-20T07:41:25.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.752+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f106c01b030 con 0x7f107c0f66a0 2021-11-20T07:41:25.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.752+0000 7f1073fff700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 0x7f106805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:25.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.752+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f106c0760e0 con 0x7f107c0f66a0 2021-11-20T07:41:25.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.752+0000 7f107affd700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 0x7f106805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:25.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.753+0000 7f107affd700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 0x7f106805e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1064004d80 tx=0x7f106400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:25.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.756+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f106c048aa0 con 0x7f107c0f66a0 2021-11-20T07:41:25.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:25 smithi112 conmon[36062]: audit 2021-11-20T07:41:24.612389+0000 mon.smithi071 (mon.0) 744 : audit [DBG] from='client.? 172.21.15.71:0/3071503899' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:25.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:25 smithi071 conmon[33687]: audit 2021-11-20T07:41:24.612389+0000 mon.smithi071 ( 2021-11-20T07:41:25.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:25 smithi071 conmon[33687]: mon.0) 744 : audit [DBG] from='client.? 172.21.15.71:0/3071503899' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:25.944 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.942+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107c0fee30 con 0x7f107c0f66a0 2021-11-20T07:41:25.944 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.943+0000 7f1073fff700 1 -- 172.21.15.71:0/3578370751 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f106c019090 con 0x7f107c0f66a0 2021-11-20T07:41:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:25.945 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:25.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 msgr2=0x7f106805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:25.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 0x7f106805e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1064004d80 tx=0x7f106400b040).stop 2021-11-20T07:41:25.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 msgr2=0x7f107c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:25.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f106c00e7c0 tx=0x7f106c00b040).stop 2021-11-20T07:41:25.947 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 shutdown_connections 2021-11-20T07:41:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f106805c2a0 0x7f106805e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f107c0f66a0 0x7f107c106df0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 --2- 172.21.15.71:0/3578370751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f107c0f7940 0x7f107c1072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 >> 172.21.15.71:0/3578370751 conn(0x7f107c0f1e00 msgr2=0x7f107c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 shutdown_connections 2021-11-20T07:41:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:25.944+0000 7f1081f89700 1 -- 172.21.15.71:0/3578370751 wait complete. 2021-11-20T07:41:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:25.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:25.958 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:26.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:26 smithi112 conmon[36062]: audit 2021-11-20T07:41:25.943964+0000 mon.smithi071 (mon.0) 745 : audit [DBG] from='client.? 172.21.15.71:0/3578370751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:26 smithi112 conmon[36062]: cluster 2021-11-20T07:41: 2021-11-20T07:41:26.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:26 smithi112 conmon[36062]: 26.407037+0000 mgr.smithi071.ynqmwn (mgr.14162) 284 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:26 smithi071 conmon[33687]: audit 2021-11-20T07:41:25.943964+0000 mon.smithi071 (mon.0 2021-11-20T07:41:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:26 smithi071 conmon[33687]: ) 745 : audit [DBG] from='client.? 172.21.15.71:0/3578370751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:26 smithi071 conmon[33687]: cluster 2021-11-20T07:41:26.407037+0000 mgr.smithi071.ynqmwn (mgr.14162) 284 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:26.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:27.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3140728370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 msgr2=0x7f7b240f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:27.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3140728370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5dd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7b180099e0 tx=0x7f7b18004d10).stop 2021-11-20T07:41:27.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3140728370 shutdown_connections 2021-11-20T07:41:27.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3140728370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5dd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3140728370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 0x7f7b240f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3140728370 >> 172.21.15.71:0/3140728370 conn(0x7f7b240f04a0 msgr2=0x7f7b240f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3140728370 shutdown_connections 2021-11-20T07:41:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3140728370 wait complete. 2021-11-20T07:41:27.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 Processor -- start 2021-11-20T07:41:27.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.085+0000 7f7b2abc6700 1 -- start start 2021-11-20T07:41:27.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:27.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 0x7f7b2410bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:27.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2410c440 con 0x7f7b240f5950 2021-11-20T07:41:27.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2410c580 con 0x7f7b240f6ca0 2021-11-20T07:41:27.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:27.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48918/0 (socket says 172.21.15.71:48918) 2021-11-20T07:41:27.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 -- 172.21.15.71:0/3022152882 learned_addr learned my addr 172.21.15.71:0/3022152882 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:27.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b23fff700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 0x7f7b2410bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:27.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 -- 172.21.15.71:0/3022152882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 msgr2=0x7f7b2410bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:27.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 0x7f7b2410bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:27.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 -- 172.21.15.71:0/3022152882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b18009640 con 0x7f7b240f5950 2021-11-20T07:41:27.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b28962700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7b140093f0 tx=0x7f7b14006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:27.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b14009ea0 con 0x7f7b240f5950 2021-11-20T07:41:27.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b14013af0 con 0x7f7b240f5950 2021-11-20T07:41:27.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2410c810 con 0x7f7b240f5950 2021-11-20T07:41:27.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b1400dad0 con 0x7f7b240f5950 2021-11-20T07:41:27.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.086+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2410cd50 con 0x7f7b240f5950 2021-11-20T07:41:27.099 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.087+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b24027a00 con 0x7f7b240f5950 2021-11-20T07:41:27.099 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.088+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7b1401a060 con 0x7f7b240f5950 2021-11-20T07:41:27.100 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.089+0000 7f7b217fa700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 0x7f7b0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:27.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.089+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7b14076680 con 0x7f7b240f5950 2021-11-20T07:41:27.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.089+0000 7f7b23fff700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 0x7f7b0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:27.102 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.089+0000 7f7b23fff700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 0x7f7b0c05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7b1800b3b0 tx=0x7f7b18004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:27.102 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.092+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b14049080 con 0x7f7b240f5950 2021-11-20T07:41:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.274+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b240fdee0 con 0x7f7b240f5950 2021-11-20T07:41:27.277 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.275+0000 7f7b217fa700 1 -- 172.21.15.71:0/3022152882 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f7b1404c6a0 con 0x7f7b240f5950 2021-11-20T07:41:27.278 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:27.278 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:27.280 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 msgr2=0x7f7b0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:27.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 0x7f7b0c05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7b1800b3b0 tx=0x7f7b18004d10).stop 2021-11-20T07:41:27.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 msgr2=0x7f7b240f5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:27.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7b140093f0 tx=0x7f7b14006ec0).stop 2021-11-20T07:41:27.281 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 shutdown_connections 2021-11-20T07:41:27.282 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7b0c05c2a0 0x7f7b0c05e760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:27.282 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b240f5950 0x7f7b240f5350 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:27.282 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 --2- 172.21.15.71:0/3022152882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b240f6ca0 0x7f7b2410bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:27.282 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 >> 172.21.15.71:0/3022152882 conn(0x7f7b240f04a0 msgr2=0x7f7b240fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:27.282 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 shutdown_connections 2021-11-20T07:41:27.283 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:27.276+0000 7f7b2abc6700 1 -- 172.21.15.71:0/3022152882 wait complete. 2021-11-20T07:41:27.283 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:27.290 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:27.291 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:27.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:27 smithi071 conmon[33687]: audit 2021-11-20T07:41:27.276152+0000 mon.smithi071 (mon.0) 746 : audit [DBG] from='client.? 172.21.15.71:0/3022152882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:27.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:27 smithi112 conmon[36062]: audit 2021-11-20T07:41:27.276152+0000 mon.smithi071 (mon.0) 746 : audit [DBG] from='client.? 172.21.15.71:0/3022152882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:28.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:28.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- 172.21.15.71:0/2614437253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 msgr2=0x7f4de00f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:28.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 --2- 172.21.15.71:0/2614437253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de00f59c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f4dd0004660 tx=0x7f4dd0009b30).stop 2021-11-20T07:41:28.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- 172.21.15.71:0/2614437253 shutdown_connections 2021-11-20T07:41:28.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 --2- 172.21.15.71:0/2614437253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4de00f6840 0x7f4de00f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:28.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 --2- 172.21.15.71:0/2614437253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de00f59c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:28.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- 172.21.15.71:0/2614437253 >> 172.21.15.71:0/2614437253 conn(0x7f4de00f0ce0 msgr2=0x7f4de00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:28.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- 172.21.15.71:0/2614437253 shutdown_connections 2021-11-20T07:41:28.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- 172.21.15.71:0/2614437253 wait complete. 2021-11-20T07:41:28.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 Processor -- start 2021-11-20T07:41:28.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.419+0000 7f4de6e75700 1 -- start start 2021-11-20T07:41:28.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:28.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4de00f6840 0x7f4de010bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:28.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010c180 con 0x7f4de00f55a0 2021-11-20T07:41:28.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010c2c0 con 0x7f4de00f6840 2021-11-20T07:41:28.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:28.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48930/0 (socket says 172.21.15.71:48930) 2021-11-20T07:41:28.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 -- 172.21.15.71:0/3388047960 learned_addr learned my addr 172.21.15.71:0/3388047960 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:28.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 -- 172.21.15.71:0/3388047960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4de00f6840 msgr2=0x7f4de010bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:28.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4de00f6840 0x7f4de010bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:28.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 -- 172.21.15.71:0/3388047960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0005040 con 0x7f4de00f55a0 2021-11-20T07:41:28.427 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de4c11700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f4dd0009f50 tx=0x7f4dd0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:28.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dd0018070 con 0x7f4de00f55a0 2021-11-20T07:41:28.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dd0013410 con 0x7f4de00f55a0 2021-11-20T07:41:28.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dd001de70 con 0x7f4de00f55a0 2021-11-20T07:41:28.428 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de010c4c0 con 0x7f4de00f55a0 2021-11-20T07:41:28.429 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.420+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de010ca20 con 0x7f4de00f55a0 2021-11-20T07:41:28.429 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.421+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de0027a00 con 0x7f4de00f55a0 2021-11-20T07:41:28.431 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.421+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4dd001d420 con 0x7f4de00f55a0 2021-11-20T07:41:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.422+0000 7f4ddd7fa700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 0x7f4dcc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.422+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4dd0077ef0 con 0x7f4de00f55a0 2021-11-20T07:41:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.422+0000 7f4ddffff700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 0x7f4dcc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.422+0000 7f4ddffff700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 0x7f4dcc05e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4dc8004a10 tx=0x7f4dc80099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.426+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dd004a800 con 0x7f4de00f55a0 2021-11-20T07:41:28.614 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.612+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de00fdd30 con 0x7f4de00f55a0 2021-11-20T07:41:28.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.613+0000 7f4ddd7fa700 1 -- 172.21.15.71:0/3388047960 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f4dd004de20 con 0x7f4de00f55a0 2021-11-20T07:41:28.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:28.616 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.614+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 msgr2=0x7f4dcc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 0x7f4dcc05e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4dc8004a10 tx=0x7f4dc80099e0).stop 2021-11-20T07:41:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 msgr2=0x7f4de005ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:28.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f4dd0009f50 tx=0x7f4dd0009b30).stop 2021-11-20T07:41:28.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 shutdown_connections 2021-11-20T07:41:28.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f4dcc05c2a0 0x7f4dcc05e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:28.618 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4de00f55a0 0x7f4de005ff20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:28.619 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 --2- 172.21.15.71:0/3388047960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4de00f6840 0x7f4de010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:28.619 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 >> 172.21.15.71:0/3388047960 conn(0x7f4de00f0ce0 msgr2=0x7f4de00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:28.619 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 shutdown_connections 2021-11-20T07:41:28.620 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:28.615+0000 7f4de6e75700 1 -- 172.21.15.71:0/3388047960 wait complete. 2021-11-20T07:41:28.620 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:28.627 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:28 smithi071 conmon[33687]: cluster 2021-11-20T07:41:28.407503+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:41:28.627 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:28 smithi071 conmon[33687]: mgr.14162) 285 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:28.629 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:28.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:28.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:28 smithi112 conmon[36062]: cluster 2021-11-20T07:41:28.407503+0000 mgr.smithi071.ynqmwn (mgr.14162) 285 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:41:29.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:29.759 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.755+0000 7fdcdf148700 1 -- 172.21.15.71:0/4116091083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 msgr2=0x7fdcd80f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:29.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.755+0000 7fdcdf148700 1 --2- 172.21.15.71:0/4116091083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd80f5910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fdcd4004660 tx=0x7fdcd4009b30).stop 2021-11-20T07:41:29.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- 172.21.15.71:0/4116091083 shutdown_connections 2021-11-20T07:41:29.760 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 --2- 172.21.15.71:0/4116091083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcd80f6790 0x7fdcd80f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:29.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 --2- 172.21.15.71:0/4116091083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd80f5910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:29.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- 172.21.15.71:0/4116091083 >> 172.21.15.71:0/4116091083 conn(0x7fdcd80f0c30 msgr2=0x7fdcd80f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:29.761 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- 172.21.15.71:0/4116091083 shutdown_connections 2021-11-20T07:41:29.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- 172.21.15.71:0/4116091083 wait complete. 2021-11-20T07:41:29.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 Processor -- start 2021-11-20T07:41:29.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- start start 2021-11-20T07:41:29.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:29.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcd80f6790 0x7fdcd8107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:29.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8107bb0 con 0x7fdcd80f54f0 2021-11-20T07:41:29.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcdf148700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8108460 con 0x7fdcd80f6790 2021-11-20T07:41:29.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcde146700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:29.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcde146700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48940/0 (socket says 172.21.15.71:48940) 2021-11-20T07:41:29.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.756+0000 7fdcde146700 1 -- 172.21.15.71:0/2212273879 learned_addr learned my addr 172.21.15.71:0/2212273879 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:29.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcde146700 1 -- 172.21.15.71:0/2212273879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcd80f6790 msgr2=0x7fdcd8107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:29.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcde146700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcd80f6790 0x7fdcd8107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:29.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcde146700 1 -- 172.21.15.71:0/2212273879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4005040 con 0x7fdcd80f54f0 2021-11-20T07:41:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcde146700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fdcd4009f50 tx=0x7fdcd4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcd4018070 con 0x7fdcd80f54f0 2021-11-20T07:41:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcd4013410 con 0x7fdcd80f54f0 2021-11-20T07:41:29.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd81086e0 con 0x7fdcd80f54f0 2021-11-20T07:41:29.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcd401c620 con 0x7fdcd80f54f0 2021-11-20T07:41:29.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.757+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd8108c40 con 0x7fdcd80f54f0 2021-11-20T07:41:29.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.758+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd80f48e0 con 0x7fdcd80f54f0 2021-11-20T07:41:29.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.758+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdcd401d420 con 0x7fdcd80f54f0 2021-11-20T07:41:29.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.758+0000 7fdcceffd700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 0x7fdcc405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:29.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.759+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdcd4078130 con 0x7fdcd80f54f0 2021-11-20T07:41:29.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.759+0000 7fdcdd945700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 0x7fdcc405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:29.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.759+0000 7fdcdd945700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 0x7fdcc405e7c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdcc8009610 tx=0x7fdcc8006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:29.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.762+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcd404aa40 con 0x7fdcd80f54f0 2021-11-20T07:41:29.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:29 smithi112 conmon[36062]: audit 2021-11-20T07:41:28.614365+0000 mon.smithi071 (mon.0) 747 : audit [DBG] from='client.? 172.21.15.71:0/3388047960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:29.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:29 smithi071 conmon[33687]: audit 2021-11-20T07:41:28.614365+0000 mon.smithi071 (mon.0) 747 : audit [DBG] from='client.? 172.21.15.71:0/3388047960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:29.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:29 smithi071 conmon[33687]: 2021-11-20T07:41:29.950 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.948+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcd8045ad0 con 0x7fdcd80f54f0 2021-11-20T07:41:29.951 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.949+0000 7fdcceffd700 1 -- 172.21.15.71:0/2212273879 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7fdcd404e060 con 0x7fdcd80f54f0 2021-11-20T07:41:29.952 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:29.952 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:29.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 msgr2=0x7fdcc405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:29.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 0x7fdcc405e7c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdcc8009610 tx=0x7fdcc8006ab0).stop 2021-11-20T07:41:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 msgr2=0x7fdcd8107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fdcd4009f50 tx=0x7fdcd4009b30).stop 2021-11-20T07:41:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 shutdown_connections 2021-11-20T07:41:29.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fdcc405c300 0x7fdcc405e7c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcd80f54f0 0x7fdcd8107080 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 --2- 172.21.15.71:0/2212273879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcd80f6790 0x7fdcd8107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.950+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 >> 172.21.15.71:0/2212273879 conn(0x7fdcd80f0c30 msgr2=0x7fdcd80fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:29.956 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.951+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 shutdown_connections 2021-11-20T07:41:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:29.951+0000 7fdcdf148700 1 -- 172.21.15.71:0/2212273879 wait complete. 2021-11-20T07:41:29.957 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:29.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:29.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:30.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:30 smithi112 conmon[36062]: audit 2021-11-20T07:41:29.950366+0000 mon.smithi071 (mon.0) 748 : audit [DBG] from='client.? 172.21.15.71:0/2212273879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:30.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:30 smithi112 conmon[36062]: cluster 2021-11-20T07:41:30.407985+0000 mgr.smithi071.ynqmwn (mgr.14162) 286 : cluster [DBG] pgmap v258: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:41:30.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:30 smithi071 conmon[33687]: audit 2021-11-20T07:41:29.950366+0000 mon.smithi071 (mon.0) 748 2021-11-20T07:41:30.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:30 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/2212273879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:30.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:30 smithi071 conmon[33687]: cluster 2021-11-20T07:41:30.407985+0000 mgr.smithi071.ynqmwn (mgr.14162) 286 : cluster [DBG] pgmap v258: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:41:30.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:31.085 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/203477792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 msgr2=0x7fbc480f80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:31.086 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/203477792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc480f80a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fbc3c004660 tx=0x7fbc3c009b30).stop 2021-11-20T07:41:31.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/203477792 shutdown_connections 2021-11-20T07:41:31.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/203477792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc480f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:31.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/203477792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc480f80a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:31.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/203477792 >> 172.21.15.71:0/203477792 conn(0x7fbc480f1d10 msgr2=0x7fbc480f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:31.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/203477792 shutdown_connections 2021-11-20T07:41:31.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.080+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/203477792 wait complete. 2021-11-20T07:41:31.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 Processor -- start 2021-11-20T07:41:31.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 -- start start 2021-11-20T07:41:31.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc48106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:31.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:31.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc48107a00 con 0x7fbc480f7c80 2021-11-20T07:41:31.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4d9e9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc48108170 con 0x7fbc480f6290 2021-11-20T07:41:31.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:31.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48950/0 (socket says 172.21.15.71:48950) 2021-11-20T07:41:31.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 -- 172.21.15.71:0/976529117 learned_addr learned my addr 172.21.15.71:0/976529117 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:31.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4c9e7700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc48106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:31.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 -- 172.21.15.71:0/976529117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 msgr2=0x7fbc48106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:31.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc48106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:31.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 -- 172.21.15.71:0/976529117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c005040 con 0x7fbc480f7c80 2021-11-20T07:41:31.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc4c9e7700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc48106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:41:31.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.081+0000 7fbc47fff700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fbc3400b410 tx=0x7fbc34006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:31.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc340125f0 con 0x7fbc480f7c80 2021-11-20T07:41:31.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc48108450 con 0x7fbc480f7c80 2021-11-20T07:41:31.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc48108a10 con 0x7fbc480f7c80 2021-11-20T07:41:31.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc34012c00 con 0x7fbc480f7c80 2021-11-20T07:41:31.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc34002aa0 con 0x7fbc480f7c80 2021-11-20T07:41:31.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.082+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc48045ad0 con 0x7fbc480f7c80 2021-11-20T07:41:31.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.083+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbc3400f030 con 0x7fbc480f7c80 2021-11-20T07:41:31.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.083+0000 7fbc457fa700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 0x7fbc3805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:31.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.083+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbc34076750 con 0x7fbc480f7c80 2021-11-20T07:41:31.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.083+0000 7fbc4c9e7700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 0x7fbc3805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:31.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.084+0000 7fbc4c9e7700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 0x7fbc3805e7c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbc3c009d60 tx=0x7fbc3c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:31.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.087+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc34049000 con 0x7fbc480f7c80 2021-11-20T07:41:31.285 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.283+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc48027a00 con 0x7fbc480f7c80 2021-11-20T07:41:31.286 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.284+0000 7fbc457fa700 1 -- 172.21.15.71:0/976529117 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7fbc3404c620 con 0x7fbc480f7c80 2021-11-20T07:41:31.287 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:31.287 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:31.289 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 msgr2=0x7fbc3805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:31.289 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 0x7fbc3805e7c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbc3c009d60 tx=0x7fbc3c009b30).stop 2021-11-20T07:41:31.289 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 msgr2=0x7fbc481073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:31.289 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fbc3400b410 tx=0x7fbc34006ef0).stop 2021-11-20T07:41:31.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 shutdown_connections 2021-11-20T07:41:31.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc480f6290 0x7fbc48106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:31.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbc3805c300 0x7fbc3805e7c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:31.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 --2- 172.21.15.71:0/976529117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc480f7c80 0x7fbc481073c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:31.291 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 >> 172.21.15.71:0/976529117 conn(0x7fbc480f1d10 msgr2=0x7fbc480fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:31.291 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 shutdown_connections 2021-11-20T07:41:31.291 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:31.286+0000 7fbc4d9e9700 1 -- 172.21.15.71:0/976529117 wait complete. 2021-11-20T07:41:31.291 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:31.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:31.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:31.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:31 smithi112 conmon[36062]: audit 2021-11-20T07:41:31.285400+0000 mon.smithi071 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.71:0/976529117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:31 smithi071 conmon[33687]: audit 2021-11-20T07:41:31.285400+0000 mon.smithi071 ( 2021-11-20T07:41:31.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:31 smithi071 conmon[33687]: mon.0) 749 : audit [DBG] from='client.? 172.21.15.71:0/976529117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:32.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:32.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 -- 172.21.15.71:0/2957057832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 msgr2=0x7f8db40f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:32.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/2957057832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db40f59c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8da4004660 tx=0x7f8da4009b30).stop 2021-11-20T07:41:32.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 -- 172.21.15.71:0/2957057832 shutdown_connections 2021-11-20T07:41:32.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/2957057832 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 0x7f8db40f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/2957057832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db40f59c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 -- 172.21.15.71:0/2957057832 >> 172.21.15.71:0/2957057832 conn(0x7f8db40f0ce0 msgr2=0x7f8db40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:32.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 -- 172.21.15.71:0/2957057832 shutdown_connections 2021-11-20T07:41:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 -- 172.21.15.71:0/2957057832 wait complete. 2021-11-20T07:41:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.435+0000 7f8dbaa30700 1 Processor -- start 2021-11-20T07:41:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8dbaa30700 1 -- start start 2021-11-20T07:41:32.442 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8dbaa30700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:32.443 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8dbaa30700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 0x7f8db410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:32.443 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8dbaa30700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db410c1b0 con 0x7f8db40f55a0 2021-11-20T07:41:32.443 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8dbaa30700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db410c2f0 con 0x7f8db40f6840 2021-11-20T07:41:32.443 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:32.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48966/0 (socket says 172.21.15.71:48966) 2021-11-20T07:41:32.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 -- 172.21.15.71:0/1211002727 learned_addr learned my addr 172.21.15.71:0/1211002727 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:32.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db37fe700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 0x7f8db410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:32.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 -- 172.21.15.71:0/1211002727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 msgr2=0x7f8db410bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:32.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 0x7f8db410bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 -- 172.21.15.71:0/1211002727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da4005040 con 0x7f8db40f55a0 2021-11-20T07:41:32.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.436+0000 7f8db3fff700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8da4009f50 tx=0x7f8da4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:32.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.437+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da4018070 con 0x7f8db40f55a0 2021-11-20T07:41:32.446 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.437+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db410c4f0 con 0x7f8db40f55a0 2021-11-20T07:41:32.446 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.437+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da4013410 con 0x7f8db40f55a0 2021-11-20T07:41:32.446 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.437+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da401de70 con 0x7f8db40f55a0 2021-11-20T07:41:32.446 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.437+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db410ca50 con 0x7f8db40f55a0 2021-11-20T07:41:32.447 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.438+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4027a00 con 0x7f8db40f55a0 2021-11-20T07:41:32.448 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.438+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8da401d420 con 0x7f8db40f55a0 2021-11-20T07:41:32.448 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.438+0000 7f8db0ff9700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 0x7f8d9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:32.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.438+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8da4077ef0 con 0x7f8db40f55a0 2021-11-20T07:41:32.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.438+0000 7f8db37fe700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 0x7f8d9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:32.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.439+0000 7f8db37fe700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 0x7f8d9c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8da80096c0 tx=0x7f8da8009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:32.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.442+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8da404a800 con 0x7f8db40f55a0 2021-11-20T07:41:32.632 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.630+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8db40fdd30 con 0x7f8db40f55a0 2021-11-20T07:41:32.633 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.631+0000 7f8db0ff9700 1 -- 172.21.15.71:0/1211002727 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f8da404de20 con 0x7f8db40f55a0 2021-11-20T07:41:32.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:32.633 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:32.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 msgr2=0x7f8d9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:32.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 0x7f8d9c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8da80096c0 tx=0x7f8da8009be0).stop 2021-11-20T07:41:32.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 msgr2=0x7f8db405ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:32.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8da4009f50 tx=0x7f8da4009b30).stop 2021-11-20T07:41:32.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 shutdown_connections 2021-11-20T07:41:32.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d9c05c2a0 0x7f8d9c05e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8db40f55a0 0x7f8db405ff20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 --2- 172.21.15.71:0/1211002727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db40f6840 0x7f8db410bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:32.637 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 >> 172.21.15.71:0/1211002727 conn(0x7f8db40f0ce0 msgr2=0x7f8db40fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:32.637 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 shutdown_connections 2021-11-20T07:41:32.637 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:32.632+0000 7f8dbaa30700 1 -- 172.21.15.71:0/1211002727 wait complete. 2021-11-20T07:41:32.639 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:32.644 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:32 smithi071 conmon[33687]: cluster 2021-11-20T07:41:32.408454+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:41:32.645 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:32 smithi071 conmon[33687]: .14162) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:41:32.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:32.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:32.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:32 smithi112 conmon[36062]: cluster 2021-11-20T07:41:32.408454+0000 mgr.smithi071.ynqmwn (mgr.14162) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T07:41:33.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:33.777 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- 172.21.15.71:0/2677621169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f6840 msgr2=0x7fead80f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 --2- 172.21.15.71:0/2677621169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f6840 0x7fead80f6ca0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7feac0004660 tx=0x7feac0009b30).stop 2021-11-20T07:41:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- 172.21.15.71:0/2677621169 shutdown_connections 2021-11-20T07:41:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 --2- 172.21.15.71:0/2677621169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f6840 0x7fead80f6ca0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:33.778 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 --2- 172.21.15.71:0/2677621169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f55a0 0x7fead80f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:33.779 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- 172.21.15.71:0/2677621169 >> 172.21.15.71:0/2677621169 conn(0x7fead80f0ce0 msgr2=0x7fead80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:33.779 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- 172.21.15.71:0/2677621169 shutdown_connections 2021-11-20T07:41:33.779 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- 172.21.15.71:0/2677621169 wait complete. 2021-11-20T07:41:33.779 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 Processor -- start 2021-11-20T07:41:33.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.773+0000 7feadd5b5700 1 -- start start 2021-11-20T07:41:33.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:33.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f6840 0x7fead810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:33.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead810c1b0 con 0x7fead80f55a0 2021-11-20T07:41:33.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead810c2f0 con 0x7fead80f6840 2021-11-20T07:41:33.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:33.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48978/0 (socket says 172.21.15.71:48978) 2021-11-20T07:41:33.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 -- 172.21.15.71:0/113740459 learned_addr learned my addr 172.21.15.71:0/113740459 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:33.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead67fc700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f6840 0x7fead810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:33.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 -- 172.21.15.71:0/113740459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f6840 msgr2=0x7fead810bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:33.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f6840 0x7fead810bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:33.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 -- 172.21.15.71:0/113740459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0005040 con 0x7fead80f55a0 2021-11-20T07:41:33.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7fead6ffd700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7feac800b790 tx=0x7feac80092b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:33.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feac800d700 con 0x7fead80f55a0 2021-11-20T07:41:33.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feac800dd10 con 0x7fead80f55a0 2021-11-20T07:41:33.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead810c5d0 con 0x7fead80f55a0 2021-11-20T07:41:33.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feac8010e10 con 0x7fead80f55a0 2021-11-20T07:41:33.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.774+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead810cb90 con 0x7fead80f55a0 2021-11-20T07:41:33.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.775+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead8027a00 con 0x7fead80f55a0 2021-11-20T07:41:33.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.776+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7feac800f410 con 0x7fead80f55a0 2021-11-20T07:41:33.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.776+0000 7feacf7fe700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 0x7feac405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:33.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.776+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7feac80765a0 con 0x7fead80f55a0 2021-11-20T07:41:33.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.776+0000 7fead67fc700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 0x7feac405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:33.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:33 smithi112 conmon[36062]: audit 2021-11-20T07:41:32.632014+0000 mon.smithi071 (mon.0) 750 : audit [DBG] from='client.? 172.21.15.71:0/1211002727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:33.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.776+0000 7fead67fc700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 0x7feac405e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7feac0013580 tx=0x7feac0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:33.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.780+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feac8048eb0 con 0x7fead80f55a0 2021-11-20T07:41:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:33 smithi071 conmon[33687]: audit 2021-11-20T07:41:32.632014+0000 mon.smithi071 (mon.0) 750 2021-11-20T07:41:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:33 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1211002727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:33.965 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.964+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fead80fddd0 con 0x7fead80f55a0 2021-11-20T07:41:33.966 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.964+0000 7feacf7fe700 1 -- 172.21.15.71:0/113740459 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7feac8016090 con 0x7fead80f55a0 2021-11-20T07:41:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:33.967 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:33.968 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 msgr2=0x7feac405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:33.968 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 0x7feac405e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7feac0013580 tx=0x7feac0009b30).stop 2021-11-20T07:41:33.968 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 msgr2=0x7fead805ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:33.969 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7feac800b790 tx=0x7feac80092b0).stop 2021-11-20T07:41:33.969 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 shutdown_connections 2021-11-20T07:41:33.969 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7feac405c2a0 0x7feac405e760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:33.969 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead80f55a0 0x7fead805ff20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:33.970 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 --2- 172.21.15.71:0/113740459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fead80f6840 0x7fead810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:33.970 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 >> 172.21.15.71:0/113740459 conn(0x7fead80f0ce0 msgr2=0x7fead80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:33.970 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 shutdown_connections 2021-11-20T07:41:33.970 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:33.966+0000 7feadd5b5700 1 -- 172.21.15.71:0/113740459 wait complete. 2021-11-20T07:41:33.971 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:33.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:33.980 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:34.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:34 smithi112 conmon[36062]: audit 2021-11-20T07:41:33.965673+0000 mon.smithi071 (mon.0) 751 : audit [DBG] from='client.? 172.21.15.71:0/113740459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:34 smithi112 conmon[36062]: cluster 2021-11-20T07:41:34.408813+0000 mgr.smithi071.ynqmwn (mgr.14162) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:41:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:34 smithi071 conmon[33687]: audit 2021-11-20T07:41:33.965673+0000 mon.smithi071 (mon. 2021-11-20T07:41:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:34 smithi071 conmon[33687]: 0) 751 : audit [DBG] from='client.? 172.21.15.71:0/113740459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:34.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:34 smithi071 conmon[33687]: cluster 2021-11-20T07:41:34.408813+0000 mgr.smithi071.ynqmwn (mgr.14162) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:41:34.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:35.111 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/604318606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 msgr2=0x7f5d580f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/604318606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d580f6ca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5d4c004660 tx=0x7f5d4c009b30).stop 2021-11-20T07:41:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/604318606 shutdown_connections 2021-11-20T07:41:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/604318606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d580f6ca0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:35.112 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/604318606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d580f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/604318606 >> 172.21.15.71:0/604318606 conn(0x7f5d580f0ce0 msgr2=0x7f5d580f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/604318606 shutdown_connections 2021-11-20T07:41:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/604318606 wait complete. 2021-11-20T07:41:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.106+0000 7f5d5e4ea700 1 Processor -- start 2021-11-20T07:41:35.113 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 -- start start 2021-11-20T07:41:35.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d5805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:35.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:35.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5810c1b0 con 0x7f5d580f6840 2021-11-20T07:41:35.114 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5810c2f0 con 0x7f5d580f55a0 2021-11-20T07:41:35.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:35.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48990/0 (socket says 172.21.15.71:48990) 2021-11-20T07:41:35.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 -- 172.21.15.71:0/364768743 learned_addr learned my addr 172.21.15.71:0/364768743 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:35.115 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d57fff700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d5805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:35.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 -- 172.21.15.71:0/364768743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 msgr2=0x7f5d5805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:35.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d5805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:35.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 -- 172.21.15.71:0/364768743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c005040 con 0x7f5d580f6840 2021-11-20T07:41:35.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d57fff700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d5805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:41:35.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d577fe700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5d4c009830 tx=0x7f5d4c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:35.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4c018070 con 0x7f5d580f6840 2021-11-20T07:41:35.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.107+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5810c570 con 0x7f5d580f6840 2021-11-20T07:41:35.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.108+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d4c013410 con 0x7f5d580f6840 2021-11-20T07:41:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.108+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4c01c620 con 0x7f5d580f6840 2021-11-20T07:41:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.108+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5810cad0 con 0x7f5d580f6840 2021-11-20T07:41:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.108+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d58027a00 con 0x7f5d580f6840 2021-11-20T07:41:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.109+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5d4c01c8e0 con 0x7f5d580f6840 2021-11-20T07:41:35.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.109+0000 7f5d54ff9700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 0x7f5d4005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:35.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.109+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5d4c078170 con 0x7f5d580f6840 2021-11-20T07:41:35.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.109+0000 7f5d57fff700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 0x7f5d4005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:35.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.110+0000 7f5d57fff700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 0x7f5d4005e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5d580f66a0 tx=0x7f5d48008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:35.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.113+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d4c04aa20 con 0x7f5d580f6840 2021-11-20T07:41:35.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.306+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d580fddd0 con 0x7f5d580f6840 2021-11-20T07:41:35.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.307+0000 7f5d54ff9700 1 -- 172.21.15.71:0/364768743 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f5d4c04e040 con 0x7f5d580f6840 2021-11-20T07:41:35.309 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:35.310 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:20.469319+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":14614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14614":{"gid":14614,"name":"cephfs.smithi071.zaejme","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.71:6829/741236990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":741236990},{"type":"v1","addr":"172.21.15.71:6829","nonce":741236990}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:35.311 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.308+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 msgr2=0x7f5d4005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:35.311 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.308+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 0x7f5d4005e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5d580f66a0 tx=0x7f5d48008040).stop 2021-11-20T07:41:35.311 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.308+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 msgr2=0x7f5d5810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:35.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.308+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5d4c009830 tx=0x7f5d4c009b30).stop 2021-11-20T07:41:35.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 shutdown_connections 2021-11-20T07:41:35.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d580f55a0 0x7f5d5805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:35.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d4005c2a0 0x7f5d4005e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:35.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 --2- 172.21.15.71:0/364768743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d580f6840 0x7f5d5810bcc0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:35.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 >> 172.21.15.71:0/364768743 conn(0x7f5d580f0ce0 msgr2=0x7f5d580fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:35.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 shutdown_connections 2021-11-20T07:41:35.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:35.309+0000 7f5d5e4ea700 1 -- 172.21.15.71:0/364768743 wait complete. 2021-11-20T07:41:35.314 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2021-11-20T07:41:35.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:20.469319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 14614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14614': {'gid': 14614, 'name': 'cephfs.smithi071.zaejme', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.71:6829/741236990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 741236990}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 741236990}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:35.324 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: audit 2021-11-20T07:41:35.136072+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: 752 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:41:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: audit 2021-11-20T07:41:35.308101+0000 mon.smithi071 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.71:0/364768743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:35.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: cluster 2021-11-20T07:41:35.365843+0000 mon.smithi071 (mon.0) 754 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:41:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: cluster 2021-11-20T07:41:35.369446+0000 mon.smithi071 (mon.0) 755 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T07:41:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: cluster 2021-11-20T07:41:35.402571+0000 mon.smithi071 (mon.0) 756 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:41:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: cluster 2021-11-20T07:41:35.405166+0000 mon.smithi071 (mon.0) 757 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:boot 2021-11-20T07:41:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: cluster 2021-11-20T07:41:35.405213+0000 mon.smithi071 (mon.0) 758 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2 up:standby 2021-11-20T07:41:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: audit 2021-11-20T07:41:35.405441+0000 mon.smithi071 (mon.0) 759 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:41:35.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:35 smithi112 conmon[36062]: audit 2021-11-20T07:41:35.514790+0000 mon.smithi071 (mon.0) 760 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:41:35.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: audit 2021-11-20T07:41:35.136072+0000 mon.smithi071 (mon. 2021-11-20T07:41:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: 0) 752 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:41:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: audit 2021-11-20T07:41:35.308101+0000 mon.smithi071 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.71:0/364768743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: cluster 2021-11-20T07:41:35.365843+0000 mon.smithi071 (mon.0) 754 : cluster [INF] daemon mds.cephfs.smithi071.zaejme restarted 2021-11-20T07:41:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: cluster 2021-11-20T07:41:35.369446+0000 mon.smithi071 (mon.0) 755 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T07:41:35.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: cluster 2021-11-20T07:41:35.402571+0000 mon.smithi071 (mon.0) 756 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:41:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: cluster 2021-11-20T07:41:35.405166+0000 mon.smithi071 (mon.0) 757 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:boot 2021-11-20T07:41:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: cluster 2021-11-20T07:41:35.405213+0000 mon.smithi071 (mon.0) 758 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2 up:standby 2021-11-20T07:41:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: audit 2021-11-20T07:41:35.405441+0000 mon.smithi071 (mon.0) 759 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.zaejme"}]: dispatch 2021-11-20T07:41:35.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:35 smithi071 conmon[33687]: audit 2021-11-20T07:41:35.514790+0000 mon.smithi071 (mon.0) 760 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:41:36.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:36.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- 172.21.15.71:0/3599646091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 msgr2=0x7fb32c0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:36.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 --2- 172.21.15.71:0/3599646091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c0f5990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fb31c004660 tx=0x7fb31c009b30).stop 2021-11-20T07:41:36.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- 172.21.15.71:0/3599646091 shutdown_connections 2021-11-20T07:41:36.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 --2- 172.21.15.71:0/3599646091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb32c0f6810 0x7fb32c0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:36.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 --2- 172.21.15.71:0/3599646091 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c0f5990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:36.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- 172.21.15.71:0/3599646091 >> 172.21.15.71:0/3599646091 conn(0x7fb32c0f0cb0 msgr2=0x7fb32c0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:36.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- 172.21.15.71:0/3599646091 shutdown_connections 2021-11-20T07:41:36.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- 172.21.15.71:0/3599646091 wait complete. 2021-11-20T07:41:36.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 Processor -- start 2021-11-20T07:41:36.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- start start 2021-11-20T07:41:36.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:36.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb32c0f6810 0x7fb32c1062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:36.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c106860 con 0x7fb32c0f5570 2021-11-20T07:41:36.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.450+0000 7fb330890700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c1069a0 con 0x7fb32c0f6810 2021-11-20T07:41:36.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:36.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49002/0 (socket says 172.21.15.71:49002) 2021-11-20T07:41:36.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 -- 172.21.15.71:0/1509990851 learned_addr learned my addr 172.21.15.71:0/1509990851 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:36.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 -- 172.21.15.71:0/1509990851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb32c0f6810 msgr2=0x7fb32c1062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:36.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb32c0f6810 0x7fb32c1062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:36.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 -- 172.21.15.71:0/1509990851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c005040 con 0x7fb32c0f5570 2021-11-20T07:41:36.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb32a59c700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb31c009f50 tx=0x7fb31c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:36.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb31c018070 con 0x7fb32c0f5570 2021-11-20T07:41:36.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c10b3e0 con 0x7fb32c0f5570 2021-11-20T07:41:36.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb31c013410 con 0x7fb32c0f5570 2021-11-20T07:41:36.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb31c01c620 con 0x7fb32c0f5570 2021-11-20T07:41:36.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.451+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c10b9a0 con 0x7fb32c0f5570 2021-11-20T07:41:36.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.452+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c027a00 con 0x7fb32c0f5570 2021-11-20T07:41:36.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.453+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb31c01d420 con 0x7fb32c0f5570 2021-11-20T07:41:36.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.453+0000 7fb31affd700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 0x7fb31405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:36.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.453+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb31c0786b0 con 0x7fb32c0f5570 2021-11-20T07:41:36.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.453+0000 7fb329d9b700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 0x7fb31405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:36.462 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.454+0000 7fb329d9b700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 0x7fb31405e7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb3200047d0 tx=0x7fb320007a40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:36.462 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.457+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb31c04afa0 con 0x7fb32c0f5570 2021-11-20T07:41:36.644 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.642+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb32c0fdd20 con 0x7fb32c0f5570 2021-11-20T07:41:36.644 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.642+0000 7fb31affd700 1 -- 172.21.15.71:0/1509990851 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2949 (secure 0 0 0) 0x7fb31c04e5c0 con 0x7fb32c0f5570 2021-11-20T07:41:36.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:36.645 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:36.408974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:resolve","state_seq":17,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 msgr2=0x7fb31405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 0x7fb31405e7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb3200047d0 tx=0x7fb320007a40).stop 2021-11-20T07:41:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 msgr2=0x7fb32c105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:36.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb31c009f50 tx=0x7fb31c009b30).stop 2021-11-20T07:41:36.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 shutdown_connections 2021-11-20T07:41:36.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb31405c2f0 0x7fb31405e7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:36.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb32c0f5570 0x7fb32c105dc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:36.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 --2- 172.21.15.71:0/1509990851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb32c0f6810 0x7fb32c1062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:36.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 >> 172.21.15.71:0/1509990851 conn(0x7fb32c0f0cb0 msgr2=0x7fb32c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:36.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 shutdown_connections 2021-11-20T07:41:36.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:36.644+0000 7fb330890700 1 -- 172.21.15.71:0/1509990851 wait complete. 2021-11-20T07:41:36.649 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 64 2021-11-20T07:41:36.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:36.408974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:resolve', 'state_seq': 17, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:36.658 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:resolve 2021-11-20T07:41:37.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:37.685 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: cluster 2021-11-20T07:41:36.409220+0000 mgr.smithi071.ynqmwn (mgr.14162) 289 : cluster 2021-11-20T07:41:37.685 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: [DBG] pgmap v262: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:41:37.685 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: cluster 2021-11-20T07:41:36.412325+0000 mon.smithi071 (mon.0) 761 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:resolve 2021-11-20T07:41:37.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:41:37.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: -20T07:41:36.412401+0000 mon.smithi071 (mon.0) 762 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2 up:standby 2021-11-20T07:41:37.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: audit 2021-11-20T07:41: 2021-11-20T07:41:37.686 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: 36.643934+0000 mon.smithi071 (mon.0) 763 : audit 2021-11-20T07:41:37.687 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:37 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/1509990851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:37.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 -- 172.21.15.71:0/1271147339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 msgr2=0x7f3bb00f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:37.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/1271147339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb00f58f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f3ba4004660 tx=0x7f3ba4009b30).stop 2021-11-20T07:41:37.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 -- 172.21.15.71:0/1271147339 shutdown_connections 2021-11-20T07:41:37.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/1271147339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 0x7f3bb00f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:37.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/1271147339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb00f58f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:37.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 -- 172.21.15.71:0/1271147339 >> 172.21.15.71:0/1271147339 conn(0x7f3bb00f0c30 msgr2=0x7f3bb00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 -- 172.21.15.71:0/1271147339 shutdown_connections 2021-11-20T07:41:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.778+0000 7f3bb4b49700 1 -- 172.21.15.71:0/1271147339 wait complete. 2021-11-20T07:41:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 Processor -- start 2021-11-20T07:41:37.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 -- start start 2021-11-20T07:41:37.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:37.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 0x7f3bb01061f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:37.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0106830 con 0x7f3bb00f54d0 2021-11-20T07:41:37.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3bb4b49700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0106fa0 con 0x7f3bb00f6770 2021-11-20T07:41:37.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3baf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:37.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3baf7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49016/0 (socket says 172.21.15.71:49016) 2021-11-20T07:41:37.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3baf7fe700 1 -- 172.21.15.71:0/2284878839 learned_addr learned my addr 172.21.15.71:0/2284878839 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:37.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.779+0000 7f3baeffd700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 0x7f3bb01061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:37.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3baf7fe700 1 -- 172.21.15.71:0/2284878839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 msgr2=0x7f3bb01061f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:37.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3baf7fe700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 0x7f3bb01061f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:37.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3baf7fe700 1 -- 172.21.15.71:0/2284878839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4005040 con 0x7f3bb00f54d0 2021-11-20T07:41:37.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3baf7fe700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3ba4009f50 tx=0x7f3ba4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:37.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba4018070 con 0x7f3bb00f54d0 2021-11-20T07:41:37.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ba4013410 con 0x7f3bb00f54d0 2021-11-20T07:41:37.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb0107220 con 0x7f3bb00f54d0 2021-11-20T07:41:37.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba401c620 con 0x7f3bb00f54d0 2021-11-20T07:41:37.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.780+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb0107780 con 0x7f3bb00f54d0 2021-11-20T07:41:37.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.781+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb0045ad0 con 0x7f3bb00f54d0 2021-11-20T07:41:37.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: cluster 2021-11-20T07:41:36.409220+0000 mgr.smithi071.ynqmwn (mgr.14162) 289 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:41:37.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: cluster 2021-11-20T07:41:36. 2021-11-20T07:41:37.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: 412325+0000 mon.smithi071 (mon.0) 761 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:resolve 2021-11-20T07:41:37.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: cluster 2021-11-20T07:41:36.412401+0000 mon.smithi071 (mon.0) 762 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2 up:standby 2021-11-20T07:41:37.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: audit 2021-11-20T07:41:36.643934+0000 mon.smithi071 (mon.0) 763 : audit [DBG] from='client.? 172.21.15.71:0/1509990851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:37.791 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:37 smithi112 conmon[36062]: 2021-11-20T07:41:37.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.781+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3ba401d420 con 0x7f3bb00f54d0 2021-11-20T07:41:37.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.781+0000 7f3b97fff700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 0x7f3b9805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:37.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.781+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3ba4078180 con 0x7f3bb00f54d0 2021-11-20T07:41:37.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.782+0000 7f3baeffd700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 0x7f3b9805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:37.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.782+0000 7f3baeffd700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 0x7f3b9805e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f3ba000b790 tx=0x7f3ba0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:37.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.785+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ba404aa40 con 0x7f3bb00f54d0 2021-11-20T07:41:37.970 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.968+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb0027a00 con 0x7f3bb00f54d0 2021-11-20T07:41:37.971 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.969+0000 7f3b97fff700 1 -- 172.21.15.71:0/2284878839 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2954 (secure 0 0 0) 0x7f3ba404e060 con 0x7f3bb00f54d0 2021-11-20T07:41:37.971 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:37.971 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:37.410761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:37.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 msgr2=0x7f3b9805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:37.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 0x7f3b9805e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f3ba000b790 tx=0x7f3ba0008040).stop 2021-11-20T07:41:37.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 msgr2=0x7f3bb0105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:37.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3ba4009f50 tx=0x7f3ba4009b30).stop 2021-11-20T07:41:37.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 shutdown_connections 2021-11-20T07:41:37.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3b9805c300 0x7f3b9805e7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:37.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bb00f54d0 0x7f3bb0105d00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:37.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 --2- 172.21.15.71:0/2284878839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bb00f6770 0x7f3bb01061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:37.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 >> 172.21.15.71:0/2284878839 conn(0x7f3bb00f0c30 msgr2=0x7f3bb00fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:37.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 shutdown_connections 2021-11-20T07:41:37.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:37.971+0000 7f3bb4b49700 1 -- 172.21.15.71:0/2284878839 wait complete. 2021-11-20T07:41:37.975 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 65 2021-11-20T07:41:37.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:37.410761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:reconnect', 'state_seq': 11, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:reconnect', 'state_seq': 18, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:37.985 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:reconnect 2021-11-20T07:41:38.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: cluster 2021-11-20T07:41:37.417225+0000 mon.smithi071 (mon.0) 764 : cluster [DBG] 2021-11-20T07:41:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:reconnect 2021-11-20T07:41:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: cluster 2021-11-20T07:41:37.417293+0000 mon.smithi071 (mon.0) 765 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:reconnect 2021-11-20T07:41:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: cluster 2021-11-20T07:41:37.417355+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:38.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: 766 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2 up:standby 2021-11-20T07:41:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: cluster 2021-11-20T07:41:37.442717+0000 mds.cephfs.smithi071.xtyomb (mds. 2021-11-20T07:41:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: 0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269995 2021-11-20T07:41:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:41:38.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: 41:37.443035+0000 mds.cephfs.smithi112.bxcrgi (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269993 2021-11-20T07:41:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: audit 2021-11-20T07:41:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: 2021-11-20T07:41:37.970301+0000 2021-11-20T07:41:38.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:38 smithi112 conmon[36062]: mon.smithi071 (mon.0) 767 : audit [DBG] from='client.? 172.21.15.71:0/2284878839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: cluster 2021-11-20T07:41:37.417225+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: 764 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:reconnect 2021-11-20T07:41:38.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: cluster 2021-11-20T07:41:37.417293+0000 mon.smithi071 (mon.0) 765 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:reconnect 2021-11-20T07:41:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: cluster 2021-11-20T07:41:37.417355+0000 mon.smithi071 (mon.0) 766 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2 up:standby 2021-11-20T07:41:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: cluster 2021-11-20T07:41:37.442717+0000 mds.cephfs.smithi071.xtyomb (mds.0) 2021-11-20T07:41:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269995 2021-11-20T07:41:38.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:41:38.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: 20T07:41:37.443035+0000 mds.cephfs.smithi112.bxcrgi (mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0269993 2021-11-20T07:41:38.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: 2021-11-20T07:41:38.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: audit 2021-11-20T07:41: 2021-11-20T07:41:38.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: 37.970301+0000 mon.smithi071 (mon.0 2021-11-20T07:41:38.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:38 smithi071 conmon[33687]: ) 767 : audit [DBG] from='client.? 172.21.15.71:0/2284878839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:38.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:39.116 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- 172.21.15.71:0/4158128650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 msgr2=0x7f30740f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:39.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 --2- 172.21.15.71:0/4158128650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f30740f5970 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f3060004660 tx=0x7f3060009b30).stop 2021-11-20T07:41:39.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- 172.21.15.71:0/4158128650 shutdown_connections 2021-11-20T07:41:39.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 --2- 172.21.15.71:0/4158128650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f30740f5970 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 --2- 172.21.15.71:0/4158128650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 0x7f30740f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- 172.21.15.71:0/4158128650 >> 172.21.15.71:0/4158128650 conn(0x7f30740f0440 msgr2=0x7f30740f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:39.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- 172.21.15.71:0/4158128650 shutdown_connections 2021-11-20T07:41:39.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- 172.21.15.71:0/4158128650 wait complete. 2021-11-20T07:41:39.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 Processor -- start 2021-11-20T07:41:39.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.111+0000 7f307aed2700 1 -- start start 2021-11-20T07:41:39.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 0x7f30740f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:39.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:39.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307410c140 con 0x7f30740f7550 2021-11-20T07:41:39.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307410c280 con 0x7f30740f6bb0 2021-11-20T07:41:39.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:39.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49028/0 (socket says 172.21.15.71:49028) 2021-11-20T07:41:39.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 -- 172.21.15.71:0/2243760898 learned_addr learned my addr 172.21.15.71:0/2243760898 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:39.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3078c6e700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 0x7f30740f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:39.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 -- 172.21.15.71:0/2243760898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 msgr2=0x7f30740f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:39.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 0x7f30740f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 -- 172.21.15.71:0/2243760898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3060005040 con 0x7f30740f7550 2021-11-20T07:41:39.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f3073fff700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f3060009830 tx=0x7f3060009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:39.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3060018070 con 0x7f30740f7550 2021-11-20T07:41:39.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307410c480 con 0x7f30740f7550 2021-11-20T07:41:39.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3060013410 con 0x7f30740f7550 2021-11-20T07:41:39.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f306001c3c0 con 0x7f30740f7550 2021-11-20T07:41:39.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.112+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307410c9e0 con 0x7f30740f7550 2021-11-20T07:41:39.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.113+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074027a00 con 0x7f30740f7550 2021-11-20T07:41:39.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.114+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f306001d420 con 0x7f30740f7550 2021-11-20T07:41:39.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.114+0000 7f30717fa700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 0x7f306405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:39.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.114+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3060077f20 con 0x7f30740f7550 2021-11-20T07:41:39.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.114+0000 7f3078c6e700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 0x7f306405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:39.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.114+0000 7f3078c6e700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 0x7f306405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f307405f7c0 tx=0x7f306800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:39.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.118+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30600a42b0 con 0x7f30740f7550 2021-11-20T07:41:39.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.340+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30740fdd40 con 0x7f30740f7550 2021-11-20T07:41:39.343 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.341+0000 7f30717fa700 1 -- 172.21.15.71:0/2243760898 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f306004a7e0 con 0x7f30740f7550 2021-11-20T07:41:39.343 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:39.343 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:38.415737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 msgr2=0x7f306405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 0x7f306405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f307405f7c0 tx=0x7f306800b040).stop 2021-11-20T07:41:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 msgr2=0x7f307410bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f3060009830 tx=0x7f3060009b30).stop 2021-11-20T07:41:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 shutdown_connections 2021-11-20T07:41:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.342+0000 7f307aed2700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30740f6bb0 0x7f30740f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.343+0000 7f307aed2700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f306405c2a0 0x7f306405e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.343+0000 7f307aed2700 1 --2- 172.21.15.71:0/2243760898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30740f7550 0x7f307410bc50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.343+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 >> 172.21.15.71:0/2243760898 conn(0x7f30740f0440 msgr2=0x7f30740fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.343+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 shutdown_connections 2021-11-20T07:41:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:39.343+0000 7f307aed2700 1 -- 172.21.15.71:0/2243760898 wait complete. 2021-11-20T07:41:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 66 2021-11-20T07:41:39.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:38.415737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:rejoin', 'state_seq': 19, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:39.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:rejoin 2021-11-20T07:41:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: cluster 2021-11-20T07:41:38.409794+0000 mgr.smithi071.ynqmwn (mgr.14162) 290 : cluster 2021-11-20T07:41:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: [DBG] pgmap v263: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:41:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: cluster 2021-11-20T07:41:38.423891+0000 mon.smithi071 (mon.0) 768 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:rejoin 2021-11-20T07:41:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: cluster 2021-11-20T07:41:38.423959+0000 mon.smithi071 (mon.0 2021-11-20T07:41:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: ) 769 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:rejoin 2021-11-20T07:41:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: cluster 2021-11-20T07:41:38.424042+0000 mon.smithi071 (mon.0) 770 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2 up:standby 2021-11-20T07:41:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: cluster 2021-11-20T07:41:38.425771+0000 mon.smithi071 (mon.0) 771 2021-11-20T07:41:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:41:39.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: audit 2021 2021-11-20T07:41:39.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:39 smithi112 conmon[36062]: -11-20T07:41:39.342144+0000 mon.smithi071 (mon.0) 772 : audit [DBG] from='client.? 172.21.15.71:0/2243760898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:39.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: cluster 2021-11-20T07:41:38.409794+0000 2021-11-20T07:41:39.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 290 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:41:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: cluster 2021-11-20T07:41:38.423891+0000 mon.smithi071 (mon.0) 768 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:rejoin 2021-11-20T07:41:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: cluster 2021-11-20T07:41:38.423959+0000 mon.smithi071 (mon.0) 769 : cluster 2021-11-20T07:41:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:rejoin 2021-11-20T07:41:39.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: cluster 2021-11-20T07:41:38.424042+0000 mon.smithi071 2021-11-20T07:41:39.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: (mon.0) 770 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2 up:standby 2021-11-20T07:41:39.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:41:39.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: 07:41:38.425771+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:39.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: 771 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:41:39.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:41:39.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: T07:41:39.342144+0000 mon.smithi071 (mon. 2021-11-20T07:41:39.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:39 smithi071 conmon[33687]: 0) 772 : audit [DBG] from='client.? 172.21.15.71:0/2243760898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:40.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:40.449 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: cluster 2021-11-20T07:41:39.428144+0000 mon.smithi071 (mon.0) 773 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:clientreplay 2021-11-20T07:41:40.450 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: 2021-11-20T07:41:40.450 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: cluster 2021-11-20T07:41:39.428195+0000 mon.smithi071 (mon.0) 774 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:active 2021-11-20T07:41:40.450 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: cluster 2021-11-20T07:41:39.428241+0000 mon.smithi071 (mon.0) 775 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:standby 2021-11-20T07:41:40.451 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: cluster 2021-11-20T07:41:39.428298+0000 mon.smithi071 (mon.0) 776 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:40.451 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:40 smithi071 conmon[33687]: cluster 2021-11-20T07:41:40.410217+0000 mgr.smithi071.ynqmwn (mgr.14162) 291 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 5.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:41:40.507 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- 172.21.15.71:0/2678393356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f5580 msgr2=0x7fcff00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:40.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 --2- 172.21.15.71:0/2678393356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f5580 0x7fcff00f59a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fcfe0004660 tx=0x7fcfe0009b30).stop 2021-11-20T07:41:40.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- 172.21.15.71:0/2678393356 shutdown_connections 2021-11-20T07:41:40.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 --2- 172.21.15.71:0/2678393356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f6820 0x7fcff00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.508 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 --2- 172.21.15.71:0/2678393356 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f5580 0x7fcff00f59a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- 172.21.15.71:0/2678393356 >> 172.21.15.71:0/2678393356 conn(0x7fcff00f0ce0 msgr2=0x7fcff00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- 172.21.15.71:0/2678393356 shutdown_connections 2021-11-20T07:41:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- 172.21.15.71:0/2678393356 wait complete. 2021-11-20T07:41:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 Processor -- start 2021-11-20T07:41:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 -- start start 2021-11-20T07:41:40.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 0x7fcff0060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:40.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.503+0000 7fcff5e73700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:40.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcff5e73700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff010c190 con 0x7fcff00f6820 2021-11-20T07:41:40.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcff5e73700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff010c2d0 con 0x7fcff00f5580 2021-11-20T07:41:40.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:40.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49040/0 (socket says 172.21.15.71:49040) 2021-11-20T07:41:40.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 -- 172.21.15.71:0/3609004659 learned_addr learned my addr 172.21.15.71:0/3609004659 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:40.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfef7fe700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 0x7fcff0060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:40.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 -- 172.21.15.71:0/3609004659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 msgr2=0x7fcff0060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:40.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 0x7fcff0060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 -- 172.21.15.71:0/3609004659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0005040 con 0x7fcff00f6820 2021-11-20T07:41:40.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfef7fe700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 0x7fcff0060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:41:40.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfeeffd700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fcfe4002700 tx=0x7fcfe4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:40.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe4009320 con 0x7fcff00f6820 2021-11-20T07:41:40.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfe400f040 con 0x7fcff00f6820 2021-11-20T07:41:40.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff010c530 con 0x7fcff00f6820 2021-11-20T07:41:40.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe40148a0 con 0x7fcff00f6820 2021-11-20T07:41:40.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.504+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff010caf0 con 0x7fcff00f6820 2021-11-20T07:41:40.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.505+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff0027a00 con 0x7fcff00f6820 2021-11-20T07:41:40.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.505+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcfe40077b0 con 0x7fcff00f6820 2021-11-20T07:41:40.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.506+0000 7fcfd7fff700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 0x7fcfd805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:40.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.506+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fcfe40768b0 con 0x7fcff00f6820 2021-11-20T07:41:40.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.506+0000 7fcfef7fe700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 0x7fcfd805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:40.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.506+0000 7fcfef7fe700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 0x7fcfd805e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fcfe0004e30 tx=0x7fcfe0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:40.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.511+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe4049170 con 0x7fcff00f6820 2021-11-20T07:41:40.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.692+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcff00fdd10 con 0x7fcff00f6820 2021-11-20T07:41:40.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.693+0000 7fcfd7fff700 1 -- 172.21.15.71:0/3609004659 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7fcfe404c790 con 0x7fcff00f6820 2021-11-20T07:41:40.696 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:40.696 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:40.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 msgr2=0x7fcfd805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 0x7fcfd805e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fcfe0004e30 tx=0x7fcfe0009b30).stop 2021-11-20T07:41:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 msgr2=0x7fcff010bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fcfe4002700 tx=0x7fcfe4009de0).stop 2021-11-20T07:41:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 shutdown_connections 2021-11-20T07:41:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcff00f5580 0x7fcff0060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfd805c2a0 0x7fcfd805e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 --2- 172.21.15.71:0/3609004659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcff00f6820 0x7fcff010bca0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 >> 172.21.15.71:0/3609004659 conn(0x7fcff00f0ce0 msgr2=0x7fcff00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 shutdown_connections 2021-11-20T07:41:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:40.695+0000 7fcff5e73700 1 -- 172.21.15.71:0/3609004659 wait complete. 2021-11-20T07:41:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:40.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:40.710 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: cluster 2021-11-20T07:41:39.428144+0000 mon.smithi071 (mon.0 2021-11-20T07:41:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: ) 773 : cluster [DBG] mds.? [v2:172.21.15.71:6826/1534235240,v1:172.21.15.71:6827/1534235240] up:clientreplay 2021-11-20T07:41:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: cluster 2021-11-20T07:41:39.428195+0000 mon.smithi071 (mon.0) 774 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2992700860,v1:172.21.15.112:6825/2992700860] up:active 2021-11-20T07:41:40.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:41:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: 41:39.428241+0000 mon.smithi071 (mon.0) 775 : cluster [DBG] 2021-11-20T07:41:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:standby 2021-11-20T07:41:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: cluster 2021-11-20T07:41:39. 2021-11-20T07:41:40.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: 428298+0000 mon.smithi071 (mon.0) 776 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.xtyomb=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:40.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: cluster 2021-11-20T07:41:40.410217 2021-11-20T07:41:40.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: +0000 mgr.smithi071.ynqmwn (mgr.14162) 291 : cluster [DBG] 2021-11-20T07:41:40.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:40 smithi112 conmon[36062]: pgmap v264: 65 pgs: 65 active+clean; 5.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:41:41.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:41.740 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:41 smithi071 conmon[33687]: audit 2021-11-20T07:41:40.694136+0000 mon.smithi071 (mon.0) 777 : audit [DBG] 2021-11-20T07:41:41.740 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:41 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/3609004659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:41 smithi112 conmon[36062]: audit 2021-11-20T07:41:40.694136+0000 mon.smithi071 (mon.0) 777 : audit [DBG] from='client.? 172.21.15.71:0/3609004659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:41.841 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 -- 172.21.15.71:0/3393260159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 msgr2=0x7f62f00f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:41.841 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3393260159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f00f59c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f62e0004660 tx=0x7f62e0009b30).stop 2021-11-20T07:41:41.841 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 -- 172.21.15.71:0/3393260159 shutdown_connections 2021-11-20T07:41:41.841 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3393260159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f00f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:41.842 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3393260159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f00f59c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:41.842 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 -- 172.21.15.71:0/3393260159 >> 172.21.15.71:0/3393260159 conn(0x7f62f00f0ce0 msgr2=0x7f62f00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:41.842 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 -- 172.21.15.71:0/3393260159 shutdown_connections 2021-11-20T07:41:41.842 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 -- 172.21.15.71:0/3393260159 wait complete. 2021-11-20T07:41:41.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.838+0000 7f62f5d35700 1 Processor -- start 2021-11-20T07:41:41.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 -- start start 2021-11-20T07:41:41.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:41.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f01062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:41.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0106870 con 0x7f62f00f55a0 2021-11-20T07:41:41.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f01069b0 con 0x7f62f00f6840 2021-11-20T07:41:41.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:41.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49048/0 (socket says 172.21.15.71:49048) 2021-11-20T07:41:41.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 -- 172.21.15.71:0/3528486412 learned_addr learned my addr 172.21.15.71:0/3528486412 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:41.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62e6ffd700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f01062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:41.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 -- 172.21.15.71:0/3528486412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 msgr2=0x7f62f01062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:41.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f01062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:41.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 -- 172.21.15.71:0/3528486412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0005040 con 0x7f62f00f55a0 2021-11-20T07:41:41.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62e6ffd700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f01062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:41:41.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ef7fe700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f62e0009f50 tx=0x7f62e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:41.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62e0018070 con 0x7f62f00f55a0 2021-11-20T07:41:41.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62e0013410 con 0x7f62f00f55a0 2021-11-20T07:41:41.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f010b3f0 con 0x7f62f00f55a0 2021-11-20T07:41:41.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62e001c620 con 0x7f62f00f55a0 2021-11-20T07:41:41.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.839+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f010b9b0 con 0x7f62f00f55a0 2021-11-20T07:41:41.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.840+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f0027a00 con 0x7f62f00f55a0 2021-11-20T07:41:41.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.841+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62e001d420 con 0x7f62f00f55a0 2021-11-20T07:41:41.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.841+0000 7f62ecff9700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 0x7f62dc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:41.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.841+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f62e0078130 con 0x7f62f00f55a0 2021-11-20T07:41:41.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.841+0000 7f62e6ffd700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 0x7f62dc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:41.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.842+0000 7f62e6ffd700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 0x7f62dc05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f62d8004a70 tx=0x7f62d8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:41.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:41.845+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62e004a9f0 con 0x7f62f00f55a0 2021-11-20T07:41:42.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.029+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62f00fdd30 con 0x7f62f00f55a0 2021-11-20T07:41:42.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.029+0000 7f62ecff9700 1 -- 172.21.15.71:0/3528486412 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f62e004e010 con 0x7f62f00f55a0 2021-11-20T07:41:42.030 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:42.031 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:42.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 msgr2=0x7f62dc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:42.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 0x7f62dc05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f62d8004a70 tx=0x7f62d8009040).stop 2021-11-20T07:41:42.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 msgr2=0x7f62f0105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:42.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f62e0009f50 tx=0x7f62e0009b30).stop 2021-11-20T07:41:42.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 shutdown_connections 2021-11-20T07:41:42.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f62dc05c2a0 0x7f62dc05e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:42.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62f00f55a0 0x7f62f0105dd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:42.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 --2- 172.21.15.71:0/3528486412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f00f6840 0x7f62f01062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:42.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 >> 172.21.15.71:0/3528486412 conn(0x7f62f00f0ce0 msgr2=0x7f62f00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:42.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 shutdown_connections 2021-11-20T07:41:42.035 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:42.031+0000 7f62f5d35700 1 -- 172.21.15.71:0/3528486412 wait complete. 2021-11-20T07:41:42.035 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:42.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:42.043 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:42 smithi112 conmon[36062]: audit 2021-11-20T07:41:42.029527+0000 mon.smithi071 (mon.0) 778 : audit [DBG] from='client.? 172.21.15.71:0/3528486412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:42 smithi112 conmon[36062]: cluster 2021-11-20T07:41:42 2021-11-20T07:41:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:42 smithi112 conmon[36062]: .410754+0000 mgr.smithi071.ynqmwn (mgr.14162) 292 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:42 smithi071 conmon[33687]: audit 2021-11-20T07:41:42.029527+0000 mon.smithi071 2021-11-20T07:41:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:42 smithi071 conmon[33687]: (mon.0) 778 : audit [DBG] from='client.? 172.21.15.71:0/3528486412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:42.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:42 smithi071 conmon[33687]: cluster 2021-11-20T07:41:42.410754+0000 mgr.smithi071.ynqmwn (mgr.14162) 292 : cluster 2021-11-20T07:41:42.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:42 smithi071 conmon[33687]: [DBG] pgmap v265: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:43.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:43.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 -- 172.21.15.71:0/1600780093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f6840 msgr2=0x7f227c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:43.173 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 --2- 172.21.15.71:0/1600780093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f6840 0x7f227c0f6ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f2270004660 tx=0x7f2270009b30).stop 2021-11-20T07:41:43.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 -- 172.21.15.71:0/1600780093 shutdown_connections 2021-11-20T07:41:43.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 --2- 172.21.15.71:0/1600780093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f6840 0x7f227c0f6ca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 --2- 172.21.15.71:0/1600780093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f55a0 0x7f227c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 -- 172.21.15.71:0/1600780093 >> 172.21.15.71:0/1600780093 conn(0x7f227c0f0ce0 msgr2=0x7f227c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 -- 172.21.15.71:0/1600780093 shutdown_connections 2021-11-20T07:41:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.170+0000 7f228460f700 1 -- 172.21.15.71:0/1600780093 wait complete. 2021-11-20T07:41:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 Processor -- start 2021-11-20T07:41:43.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 -- start start 2021-11-20T07:41:43.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:43.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f6840 0x7f227c10bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:43.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c10c180 con 0x7f227c0f55a0 2021-11-20T07:41:43.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f228460f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c10c2c0 con 0x7f227c0f6840 2021-11-20T07:41:43.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49060/0 (socket says 172.21.15.71:49060) 2021-11-20T07:41:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 -- 172.21.15.71:0/813052411 learned_addr learned my addr 172.21.15.71:0/813052411 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f2281baa700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f6840 0x7f227c10bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 -- 172.21.15.71:0/813052411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f6840 msgr2=0x7f227c10bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f6840 0x7f227c10bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 -- 172.21.15.71:0/813052411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2270005040 con 0x7f227c0f55a0 2021-11-20T07:41:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.171+0000 7f22823ab700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f226c00d820 tx=0x7f226c004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.172+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226c00b510 con 0x7f227c0f55a0 2021-11-20T07:41:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.172+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f226c00f040 con 0x7f227c0f55a0 2021-11-20T07:41:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.172+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c10c520 con 0x7f227c0f55a0 2021-11-20T07:41:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.172+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226c013a30 con 0x7f227c0f55a0 2021-11-20T07:41:43.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.172+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c10cae0 con 0x7f227c0f55a0 2021-11-20T07:41:43.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.173+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c027a00 con 0x7f227c0f55a0 2021-11-20T07:41:43.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.173+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f226c01a030 con 0x7f227c0f55a0 2021-11-20T07:41:43.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.173+0000 7f227affd700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 0x7f226405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:43.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.173+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f226c0760a0 con 0x7f227c0f55a0 2021-11-20T07:41:43.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.173+0000 7f2281baa700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 0x7f226405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:43.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.174+0000 7f2281baa700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 0x7f226405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f2270004e30 tx=0x7f2270009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:43.184 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.177+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f226c0489e0 con 0x7f227c0f55a0 2021-11-20T07:41:43.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.361+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f227c0fdd30 con 0x7f227c0f55a0 2021-11-20T07:41:43.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.362+0000 7f227affd700 1 -- 172.21.15.71:0/813052411 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f226c018090 con 0x7f227c0f55a0 2021-11-20T07:41:43.363 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:43.364 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:43.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 msgr2=0x7f226405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:43.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 0x7f226405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f2270004e30 tx=0x7f2270009b30).stop 2021-11-20T07:41:43.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 msgr2=0x7f227c05ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:43.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f226c00d820 tx=0x7f226c004e80).stop 2021-11-20T07:41:43.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 shutdown_connections 2021-11-20T07:41:43.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f226405c2a0 0x7f226405e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f227c0f55a0 0x7f227c05ff20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 --2- 172.21.15.71:0/813052411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f227c0f6840 0x7f227c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:43.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 >> 172.21.15.71:0/813052411 conn(0x7f227c0f0ce0 msgr2=0x7f227c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:43.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 shutdown_connections 2021-11-20T07:41:43.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:43.363+0000 7f228460f700 1 -- 172.21.15.71:0/813052411 wait complete. 2021-11-20T07:41:43.367 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:43.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:43.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:43.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:43 smithi112 conmon[36062]: audit 2021-11-20T07:41:43.362131+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:43.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:43 smithi112 conmon[36062]: 779 : audit [DBG] from='client.? 172.21.15.71:0/813052411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:43.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:43 smithi071 conmon[33687]: audit 2021-11-20T07:41:43.362131+0000 mon.smithi071 (mon. 2021-11-20T07:41:43.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:43 smithi071 conmon[33687]: 0) 779 : audit [DBG] from='client.? 172.21.15.71:0/813052411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:44.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:44.509 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- 172.21.15.71:0/849604190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f55a0 msgr2=0x7f8a380f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:44.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/849604190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f55a0 0x7f8a380f59c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f8a34004660 tx=0x7f8a34009b30).stop 2021-11-20T07:41:44.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- 172.21.15.71:0/849604190 shutdown_connections 2021-11-20T07:41:44.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/849604190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f6840 0x7f8a380f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.510 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/849604190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f55a0 0x7f8a380f59c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- 172.21.15.71:0/849604190 >> 172.21.15.71:0/849604190 conn(0x7f8a380f0ce0 msgr2=0x7f8a380f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:44.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- 172.21.15.71:0/849604190 shutdown_connections 2021-11-20T07:41:44.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- 172.21.15.71:0/849604190 wait complete. 2021-11-20T07:41:44.511 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 Processor -- start 2021-11-20T07:41:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.505+0000 7f8a3f71a700 1 -- start start 2021-11-20T07:41:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3f71a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f55a0 0x7f8a3805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3f71a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3f71a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3810c1b0 con 0x7f8a380f6840 2021-11-20T07:41:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3f71a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3810c2f0 con 0x7f8a380f55a0 2021-11-20T07:41:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49074/0 (socket says 172.21.15.71:49074) 2021-11-20T07:41:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 -- 172.21.15.71:0/4225377885 learned_addr learned my addr 172.21.15.71:0/4225377885 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3d4b6700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f55a0 0x7f8a3805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 -- 172.21.15.71:0/4225377885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f55a0 msgr2=0x7f8a3805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f55a0 0x7f8a3805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 -- 172.21.15.71:0/4225377885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34005040 con 0x7f8a380f6840 2021-11-20T07:41:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3ccb5700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8a28002700 tx=0x7f8a28009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a28009320 con 0x7f8a380f6840 2021-11-20T07:41:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a2800f040 con 0x7f8a380f6840 2021-11-20T07:41:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3810c550 con 0x7f8a380f6840 2021-11-20T07:41:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.506+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a280148a0 con 0x7f8a380f6840 2021-11-20T07:41:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.507+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3810cb10 con 0x7f8a380f6840 2021-11-20T07:41:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.507+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a38027a00 con 0x7f8a380f6840 2021-11-20T07:41:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.508+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8a280077b0 con 0x7f8a380f6840 2021-11-20T07:41:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.508+0000 7f8a2dffb700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 0x7f8a2405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.508+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f8a280768b0 con 0x7f8a380f6840 2021-11-20T07:41:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.508+0000 7f8a3d4b6700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 0x7f8a2405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:44.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.509+0000 7f8a3d4b6700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 0x7f8a2405e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8a34004e30 tx=0x7f8a34009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:44.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.512+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a28049170 con 0x7f8a380f6840 2021-11-20T07:41:44.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:44 smithi071 conmon[33687]: cluster 2021-11-20T07:41:44.411215+0000 mgr.smithi071.ynqmwn (mgr.14162) 293 : 2021-11-20T07:41:44.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:44 smithi071 conmon[33687]: cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:44.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.697+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a380fdd30 con 0x7f8a380f6840 2021-11-20T07:41:44.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.698+0000 7f8a2dffb700 1 -- 172.21.15.71:0/4225377885 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f8a2804c790 con 0x7f8a380f6840 2021-11-20T07:41:44.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:44.700 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:44.701 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 msgr2=0x7f8a2405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:44.702 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 0x7f8a2405e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8a34004e30 tx=0x7f8a34009b30).stop 2021-11-20T07:41:44.702 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 msgr2=0x7f8a3810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:44.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8a28002700 tx=0x7f8a28009de0).stop 2021-11-20T07:41:44.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 shutdown_connections 2021-11-20T07:41:44.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a380f55a0 0x7f8a3805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8a2405c2a0 0x7f8a2405e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.699+0000 7f8a3f71a700 1 --2- 172.21.15.71:0/4225377885 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a380f6840 0x7f8a3810bcc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:44.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.700+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 >> 172.21.15.71:0/4225377885 conn(0x7f8a380f0ce0 msgr2=0x7f8a380fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:44.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.700+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 shutdown_connections 2021-11-20T07:41:44.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:44.700+0000 7f8a3f71a700 1 -- 172.21.15.71:0/4225377885 wait complete. 2021-11-20T07:41:44.706 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:44.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:44.712 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:44.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:44 smithi112 conmon[36062]: cluster 2021-11-20T07:41:44.411215+0000 mgr.smithi071.ynqmwn (mgr.14162) 293 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:41:45.713 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:45.741 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:45 smithi071 conmon[33687]: audit 2021-11-20T07:41:44.698174+0000 mon.smithi071 (mon.0) 780 : audit [DBG] from='client.? 172.21.15.71:0/4225377885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:45.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:45 smithi112 conmon[36062]: audit 2021-11-20T07:41:44.698174+0000 mon.smithi071 (mon.0) 780 : audit [DBG] from='client.? 172.21.15.71:0/4225377885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:45.842 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 -- 172.21.15.71:0/3635224449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f4840 msgr2=0x7fea400f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 --2- 172.21.15.71:0/3635224449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f4840 0x7fea400f4c60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fea3c004660 tx=0x7fea3c009b30).stop 2021-11-20T07:41:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 -- 172.21.15.71:0/3635224449 shutdown_connections 2021-11-20T07:41:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 --2- 172.21.15.71:0/3635224449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea400f51e0 0x7fea400f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 --2- 172.21.15.71:0/3635224449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f4840 0x7fea400f4c60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.837+0000 7fea46154700 1 -- 172.21.15.71:0/3635224449 >> 172.21.15.71:0/3635224449 conn(0x7fea400f0360 msgr2=0x7fea400f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 -- 172.21.15.71:0/3635224449 shutdown_connections 2021-11-20T07:41:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 -- 172.21.15.71:0/3635224449 wait complete. 2021-11-20T07:41:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 Processor -- start 2021-11-20T07:41:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 -- start start 2021-11-20T07:41:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea400f4840 0x7fea40106e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea401079b0 con 0x7fea400f51e0 2021-11-20T07:41:45.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea46154700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40108130 con 0x7fea400f4840 2021-11-20T07:41:45.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea44951700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:45.847 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea44951700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49084/0 (socket says 172.21.15.71:49084) 2021-11-20T07:41:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.838+0000 7fea44951700 1 -- 172.21.15.71:0/1559606976 learned_addr learned my addr 172.21.15.71:0/1559606976 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea44951700 1 -- 172.21.15.71:0/1559606976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea400f4840 msgr2=0x7fea40106e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea44951700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea400f4840 0x7fea40106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea44951700 1 -- 172.21.15.71:0/1559606976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c005040 con 0x7fea400f51e0 2021-11-20T07:41:45.848 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea44951700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fea300093f0 tx=0x7fea30006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:45.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea30009ea0 con 0x7fea400f51e0 2021-11-20T07:41:45.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea30013af0 con 0x7fea400f51e0 2021-11-20T07:41:45.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea401083b0 con 0x7fea400f51e0 2021-11-20T07:41:45.849 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea3000dad0 con 0x7fea400f51e0 2021-11-20T07:41:45.850 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.839+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea40108970 con 0x7fea400f51e0 2021-11-20T07:41:45.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.840+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea40045ad0 con 0x7fea400f51e0 2021-11-20T07:41:45.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.840+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fea3001a060 con 0x7fea400f51e0 2021-11-20T07:41:45.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.841+0000 7fea35ffb700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 0x7fea2c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:45.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.841+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fea300762f0 con 0x7fea400f51e0 2021-11-20T07:41:45.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.841+0000 7fea45152700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 0x7fea2c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:45.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.841+0000 7fea45152700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 0x7fea2c05e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fea3c009f50 tx=0x7fea3c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:45.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:45.845+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea30048bb0 con 0x7fea400f51e0 2021-11-20T07:41:46.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.028+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea40027a00 con 0x7fea400f51e0 2021-11-20T07:41:46.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.028+0000 7fea35ffb700 1 -- 172.21.15.71:0/1559606976 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7fea3004c1d0 con 0x7fea400f51e0 2021-11-20T07:41:46.030 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:46.030 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:46.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 msgr2=0x7fea2c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:46.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 0x7fea2c05e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fea3c009f50 tx=0x7fea3c009b30).stop 2021-11-20T07:41:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 msgr2=0x7fea40107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fea300093f0 tx=0x7fea30006ec0).stop 2021-11-20T07:41:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 shutdown_connections 2021-11-20T07:41:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea400f4840 0x7fea40106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fea2c05c300 0x7fea2c05e7c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 --2- 172.21.15.71:0/1559606976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea400f51e0 0x7fea40107320 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 >> 172.21.15.71:0/1559606976 conn(0x7fea400f0360 msgr2=0x7fea400fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:46.035 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 shutdown_connections 2021-11-20T07:41:46.035 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:46.030+0000 7fea46154700 1 -- 172.21.15.71:0/1559606976 wait complete. 2021-11-20T07:41:46.035 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:46.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:46.043 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:46.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:46 smithi112 conmon[36062]: audit 2021-11-20T07:41:46.028623+0000 mon.smithi071 (mon.0 2021-11-20T07:41:46.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:46 smithi112 conmon[36062]: ) 781 : audit [DBG] from='client.? 172.21.15.71:0/1559606976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:46.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:46 smithi112 conmon[36062]: cluster 2021-11-20T07:41:46.411709+0000 mgr.smithi071.ynqmwn (mgr.14162) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 370 B/s wr, 1 op/s 2021-11-20T07:41:46.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:46 smithi071 conmon[33687]: audit 2021-11-20T07:41:46.028623+0000 mon.smithi071 (mon 2021-11-20T07:41:46.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:46 smithi071 conmon[33687]: .0) 781 : audit [DBG] from='client.? 172.21.15.71:0/1559606976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:46.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:46 smithi071 conmon[33687]: cluster 2021-11-20T07:41:46.411709+0000 mgr.smithi071.ynqmwn 2021-11-20T07:41:46.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:46 smithi071 conmon[33687]: (mgr.14162) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 370 B/s wr, 1 op/s 2021-11-20T07:41:47.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:47.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- 172.21.15.71:0/2123424670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f5580 msgr2=0x7f72dc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:47.174 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 --2- 172.21.15.71:0/2123424670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f5580 0x7f72dc0f59a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f72cc004660 tx=0x7f72cc009b30).stop 2021-11-20T07:41:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- 172.21.15.71:0/2123424670 shutdown_connections 2021-11-20T07:41:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 --2- 172.21.15.71:0/2123424670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f6820 0x7f72dc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 --2- 172.21.15.71:0/2123424670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f5580 0x7f72dc0f59a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- 172.21.15.71:0/2123424670 >> 172.21.15.71:0/2123424670 conn(0x7f72dc0f0ce0 msgr2=0x7f72dc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:47.175 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- 172.21.15.71:0/2123424670 shutdown_connections 2021-11-20T07:41:47.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- 172.21.15.71:0/2123424670 wait complete. 2021-11-20T07:41:47.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 Processor -- start 2021-11-20T07:41:47.176 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.170+0000 7f72e257d700 1 -- start start 2021-11-20T07:41:47.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 0x7f72dc105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:47.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:47.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc106850 con 0x7f72dc0f6820 2021-11-20T07:41:47.177 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc106990 con 0x7f72dc0f5580 2021-11-20T07:41:47.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:47.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49094/0 (socket says 172.21.15.71:49094) 2021-11-20T07:41:47.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 -- 172.21.15.71:0/1332671582 learned_addr learned my addr 172.21.15.71:0/1332671582 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:47.178 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72dbfff700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 0x7f72dc105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 -- 172.21.15.71:0/1332671582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 msgr2=0x7f72dc105db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 0x7f72dc105db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 -- 172.21.15.71:0/1332671582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc005040 con 0x7f72dc0f6820 2021-11-20T07:41:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72dbfff700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 0x7f72dc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:41:47.179 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72db7fe700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f72c400c530 tx=0x7f72c4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:47.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72c400f650 con 0x7f72dc0f6820 2021-11-20T07:41:47.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72c400fc60 con 0x7f72dc0f6820 2021-11-20T07:41:47.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc05fb40 con 0x7f72dc0f6820 2021-11-20T07:41:47.180 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72c4012d30 con 0x7f72dc0f6820 2021-11-20T07:41:47.181 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.171+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc060100 con 0x7f72dc0f6820 2021-11-20T07:41:47.181 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.172+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc0fdd10 con 0x7f72dc0f6820 2021-11-20T07:41:47.181 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.173+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f72c4010070 con 0x7f72dc0f6820 2021-11-20T07:41:47.181 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.173+0000 7f72d8ff9700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 0x7f72c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:47.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.173+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f72c4076050 con 0x7f72dc0f6820 2021-11-20T07:41:47.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.173+0000 7f72dbfff700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 0x7f72c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:47.182 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.173+0000 7f72dbfff700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 0x7f72c805e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f72cc009d60 tx=0x7f72cc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:47.183 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.177+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72c40452b0 con 0x7f72dc0f6820 2021-11-20T07:41:47.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.359+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72dc10d580 con 0x7f72dc0f6820 2021-11-20T07:41:47.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.360+0000 7f72d8ff9700 1 -- 172.21.15.71:0/1332671582 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f72c4048910 con 0x7f72dc0f6820 2021-11-20T07:41:47.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:47.361 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:47.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 msgr2=0x7f72c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:47.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 0x7f72c805e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f72cc009d60 tx=0x7f72cc009b30).stop 2021-11-20T07:41:47.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 msgr2=0x7f72dc1062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:47.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f72c400c530 tx=0x7f72c4007e20).stop 2021-11-20T07:41:47.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 shutdown_connections 2021-11-20T07:41:47.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f72dc0f5580 0x7f72dc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f72c805c2a0 0x7f72c805e760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 --2- 172.21.15.71:0/1332671582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72dc0f6820 0x7f72dc1062a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 >> 172.21.15.71:0/1332671582 conn(0x7f72dc0f0ce0 msgr2=0x7f72dc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 shutdown_connections 2021-11-20T07:41:47.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:47.361+0000 7f72e257d700 1 -- 172.21.15.71:0/1332671582 wait complete. 2021-11-20T07:41:47.365 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:47.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:47.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:47.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:47 smithi112 conmon[36062]: audit 2021-11-20T07:41:47.359910+0000 2021-11-20T07:41:47.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:47 smithi112 conmon[36062]: mon.smithi071 (mon.0) 782 : audit [DBG] from='client.? 172.21.15.71:0/1332671582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:47.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:47 smithi071 conmon[33687]: audit 2021-11-20T07:41:47.359910+0000 mon.smithi071 (mon. 2021-11-20T07:41:47.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:47 smithi071 conmon[33687]: 0) 782 : audit [DBG] from='client.? 172.21.15.71:0/1332671582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:48.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:48.516 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- 172.21.15.71:0/495925594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 msgr2=0x7f1d9c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:48.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 --2- 172.21.15.71:0/495925594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c0f4be0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f1d8c004660 tx=0x7f1d8c009b30).stop 2021-11-20T07:41:48.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- 172.21.15.71:0/495925594 shutdown_connections 2021-11-20T07:41:48.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 --2- 172.21.15.71:0/495925594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c0f4be0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.517 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 --2- 172.21.15.71:0/495925594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 0x7f1d9c0f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.518 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- 172.21.15.71:0/495925594 >> 172.21.15.71:0/495925594 conn(0x7f1d9c0f0440 msgr2=0x7f1d9c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:48.518 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- 172.21.15.71:0/495925594 shutdown_connections 2021-11-20T07:41:48.518 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- 172.21.15.71:0/495925594 wait complete. 2021-11-20T07:41:48.518 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 Processor -- start 2021-11-20T07:41:48.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- start start 2021-11-20T07:41:48.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:48.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 0x7f1d9c10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:48.519 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c10df10 con 0x7f1d9c0f4760 2021-11-20T07:41:48.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da44f0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c10e050 con 0x7f1d9c0f5a10 2021-11-20T07:41:48.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da228c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:48.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da228c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49104/0 (socket says 172.21.15.71:49104) 2021-11-20T07:41:48.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.512+0000 7f1da228c700 1 -- 172.21.15.71:0/1418753824 learned_addr learned my addr 172.21.15.71:0/1418753824 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:48.520 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da1a8b700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 0x7f1d9c10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:48.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da228c700 1 -- 172.21.15.71:0/1418753824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 msgr2=0x7f1d9c10d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:48.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da228c700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 0x7f1d9c10d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da228c700 1 -- 172.21.15.71:0/1418753824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c005040 con 0x7f1d9c0f4760 2021-11-20T07:41:48.521 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da228c700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f1d9800c510 tx=0x7f1d98007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:48.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d9800f650 con 0x7f1d9c0f4760 2021-11-20T07:41:48.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d9800fc60 con 0x7f1d9c0f4760 2021-11-20T07:41:48.522 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c112af0 con 0x7f1d9c0f4760 2021-11-20T07:41:48.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d98011c00 con 0x7f1d9c0f4760 2021-11-20T07:41:48.523 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.513+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c113030 con 0x7f1d9c0f4760 2021-11-20T07:41:48.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.514+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c027a00 con 0x7f1d9c0f4760 2021-11-20T07:41:48.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.514+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1d98010070 con 0x7f1d9c0f4760 2021-11-20T07:41:48.524 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.514+0000 7f1d92ffd700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 0x7f1d8805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:48.525 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.515+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f1d98075de0 con 0x7f1d9c0f4760 2021-11-20T07:41:48.525 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.515+0000 7f1da1a8b700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 0x7f1d8805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:48.525 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.515+0000 7f1da1a8b700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 0x7f1d8805e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1d8c013580 tx=0x7f1d8c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:48.526 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.519+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d9804c0d0 con 0x7f1d9c0f4760 2021-11-20T07:41:48.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.702+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d9c1054a0 con 0x7f1d9c0f4760 2021-11-20T07:41:48.703 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.703+0000 7f1d92ffd700 1 -- 172.21.15.71:0/1418753824 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f1d98017ba0 con 0x7f1d9c0f4760 2021-11-20T07:41:48.704 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:48.705 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:48.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 msgr2=0x7f1d8805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:48.706 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 0x7f1d8805e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1d8c013580 tx=0x7f1d8c009b30).stop 2021-11-20T07:41:48.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 msgr2=0x7f1d9c10d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:48.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f1d9800c510 tx=0x7f1d98007e00).stop 2021-11-20T07:41:48.707 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 shutdown_connections 2021-11-20T07:41:48.709 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f1d8805c2a0 0x7f1d8805e760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.709 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c0f4760 0x7f1d9c10d470 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.710 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 --2- 172.21.15.71:0/1418753824 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d9c0f5a10 0x7f1d9c10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:48.710 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 >> 172.21.15.71:0/1418753824 conn(0x7f1d9c0f0440 msgr2=0x7f1d9c0f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:48.710 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 shutdown_connections 2021-11-20T07:41:48.711 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:48.704+0000 7f1da44f0700 1 -- 172.21.15.71:0/1418753824 wait complete. 2021-11-20T07:41:48.711 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:48.715 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:48 smithi071 conmon[33687]: cluster 2021-11-20T07:41:48.412206+0000 mgr.smithi071.ynqmwn 2021-11-20T07:41:48.716 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:48 smithi071 conmon[33687]: (mgr.14162) 295 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:48.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:48.717 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:48 smithi112 conmon[36062]: cluster 2021-11-20T07:41:48.412206+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:41:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:48 smithi112 conmon[36062]: 295 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:49.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:49.742 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:49 smithi071 conmon[33687]: audit 2021-11-20T07:41:48.702976+0000 mon.smithi071 (mon.0) 783 : audit [DBG] from='client.? 172.21.15.71:0/1418753824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:49.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:49 smithi112 conmon[36062]: audit 2021-11-20T07:41:48.702976+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:49.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:49 smithi112 conmon[36062]: 783 : audit [DBG] from='client.? 172.21.15.71:0/1418753824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:49.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- 172.21.15.71:0/2795074876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f5a50 msgr2=0x7f2cac0f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:49.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/2795074876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f5a50 0x7f2cac0f5e70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f2ca0004660 tx=0x7f2ca0009b30).stop 2021-11-20T07:41:49.854 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- 172.21.15.71:0/2795074876 shutdown_connections 2021-11-20T07:41:49.855 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/2795074876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f6360 0x7f2cac0f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:49.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/2795074876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f5a50 0x7f2cac0f5e70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:49.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- 172.21.15.71:0/2795074876 >> 172.21.15.71:0/2795074876 conn(0x7f2cac0f0400 msgr2=0x7f2cac0f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:49.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- 172.21.15.71:0/2795074876 shutdown_connections 2021-11-20T07:41:49.856 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- 172.21.15.71:0/2795074876 wait complete. 2021-11-20T07:41:49.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 Processor -- start 2021-11-20T07:41:49.857 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.850+0000 7f2cb1f90700 1 -- start start 2021-11-20T07:41:49.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f5a50 0x7f2cac109a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:49.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:49.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac0f5540 con 0x7f2cac0f6360 2021-11-20T07:41:49.858 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac0f5680 con 0x7f2cac0f5a50 2021-11-20T07:41:49.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:49.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49116/0 (socket says 172.21.15.71:49116) 2021-11-20T07:41:49.859 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 -- 172.21.15.71:0/417427811 learned_addr learned my addr 172.21.15.71:0/417427811 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:49.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb0f8e700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f5a50 0x7f2cac109a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:49.860 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 -- 172.21.15.71:0/417427811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f5a50 msgr2=0x7f2cac109a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:49.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f5a50 0x7f2cac109a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:49.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 -- 172.21.15.71:0/417427811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0005040 con 0x7f2cac0f6360 2021-11-20T07:41:49.861 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cabfff700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f2c9800c790 tx=0x7f2c98007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:49.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c98002590 con 0x7f2cac0f6360 2021-11-20T07:41:49.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c98002ba0 con 0x7f2cac0f6360 2021-11-20T07:41:49.862 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cac10c4c0 con 0x7f2cac0f6360 2021-11-20T07:41:49.863 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c98013d90 con 0x7f2cac0f6360 2021-11-20T07:41:49.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.851+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cac10ca80 con 0x7f2cac0f6360 2021-11-20T07:41:49.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.852+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cac045ad0 con 0x7f2cac0f6360 2021-11-20T07:41:49.864 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.853+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2c9801a060 con 0x7f2cac0f6360 2021-11-20T07:41:49.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.854+0000 7f2ca97fa700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 0x7f2c9c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:49.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.854+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f2c980767d0 con 0x7f2cac0f6360 2021-11-20T07:41:49.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.854+0000 7f2cb0f8e700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 0x7f2c9c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:49.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.854+0000 7f2cb0f8e700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 0x7f2c9c05e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2ca0004630 tx=0x7f2ca0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:49.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:49.857+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c980490e0 con 0x7f2cac0f6360 2021-11-20T07:41:50.054 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.053+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cac0fdd40 con 0x7f2cac0f6360 2021-11-20T07:41:50.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.054+0000 7f2ca97fa700 1 -- 172.21.15.71:0/417427811 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f2c9804c700 con 0x7f2cac0f6360 2021-11-20T07:41:50.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:50.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:50.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.055+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 msgr2=0x7f2c9c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:50.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.055+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 0x7f2c9c05e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2ca0004630 tx=0x7f2ca0009b30).stop 2021-11-20T07:41:50.057 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.055+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 msgr2=0x7f2cac10bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:50.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f2c9800c790 tx=0x7f2c98007d70).stop 2021-11-20T07:41:50.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 shutdown_connections 2021-11-20T07:41:50.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cac0f5a50 0x7f2cac109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:50.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2c9c05c300 0x7f2c9c05e7c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 --2- 172.21.15.71:0/417427811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2cac0f6360 0x7f2cac10bf10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 >> 172.21.15.71:0/417427811 conn(0x7f2cac0f0400 msgr2=0x7f2cac0fa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 shutdown_connections 2021-11-20T07:41:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:50.056+0000 7f2cb1f90700 1 -- 172.21.15.71:0/417427811 wait complete. 2021-11-20T07:41:50.059 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:50.068 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:50.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:50.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:50 smithi071 conmon[33687]: audit 2021-11-20T07:41:50.054267+0000 mon.smithi071 (mon 2021-11-20T07:41:50.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:50 smithi071 conmon[33687]: .0) 784 : audit [DBG] from='client.? 172.21.15.71:0/417427811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:50.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:50 smithi071 conmon[33687]: cluster 2021-11-20T07:41:50.412547+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:41:50.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:50 smithi071 conmon[33687]: .14162) 296 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:50.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:50 smithi112 conmon[36062]: audit 2021-11-20T07:41:50.054267+0000 mon.smithi071 (mon.0) 784 : audit [DBG] 2021-11-20T07:41:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:50 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/417427811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:50 smithi112 conmon[36062]: cluster 2021-11-20T07:41:50.412547+0000 mgr.smithi071.ynqmwn (mgr.14162) 296 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:41:51.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:51.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 -- 172.21.15.71:0/729242556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 msgr2=0x7fe3400f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:51.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 --2- 172.21.15.71:0/729242556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe3400f6c80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fe33c004660 tx=0x7fe33c009b30).stop 2021-11-20T07:41:51.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 -- 172.21.15.71:0/729242556 shutdown_connections 2021-11-20T07:41:51.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 --2- 172.21.15.71:0/729242556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe3400f6c80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 --2- 172.21.15.71:0/729242556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 0x7fe3400f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 -- 172.21.15.71:0/729242556 >> 172.21.15.71:0/729242556 conn(0x7fe3400f0ce0 msgr2=0x7fe3400f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 -- 172.21.15.71:0/729242556 shutdown_connections 2021-11-20T07:41:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.197+0000 7fe3485dc700 1 -- 172.21.15.71:0/729242556 wait complete. 2021-11-20T07:41:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 Processor -- start 2021-11-20T07:41:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 -- start start 2021-11-20T07:41:51.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 0x7fe340105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:51.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:51.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3401067c0 con 0x7fe3400f6820 2021-11-20T07:41:51.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe3485dc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340106900 con 0x7fe3400f5580 2021-11-20T07:41:51.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:51.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49134/0 (socket says 172.21.15.71:49134) 2021-11-20T07:41:51.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 -- 172.21.15.71:0/323474891 learned_addr learned my addr 172.21.15.71:0/323474891 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:51.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe346378700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 0x7fe340105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:51.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 -- 172.21.15.71:0/323474891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 msgr2=0x7fe340105d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:51.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 0x7fe340105d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.198+0000 7fe345b77700 1 -- 172.21.15.71:0/323474891 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c005040 con 0x7fe3400f6820 2021-11-20T07:41:51.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe345b77700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe33c009f50 tx=0x7fe33c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:51.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33c018070 con 0x7fe3400f6820 2021-11-20T07:41:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe33c013410 con 0x7fe3400f6820 2021-11-20T07:41:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34010b340 con 0x7fe3400f6820 2021-11-20T07:41:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33c01c3c0 con 0x7fe3400f6820 2021-11-20T07:41:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.199+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34010b900 con 0x7fe3400f6820 2021-11-20T07:41:51.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.200+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe340046330 con 0x7fe3400f6820 2021-11-20T07:41:51.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.200+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe33c01d420 con 0x7fe3400f6820 2021-11-20T07:41:51.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.200+0000 7fe336ffd700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 0x7fe32c05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:51.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.200+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe33c078000 con 0x7fe3400f6820 2021-11-20T07:41:51.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.200+0000 7fe346378700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 0x7fe32c05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:51.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.201+0000 7fe346378700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 0x7fe32c05e7b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe3400f6680 tx=0x7fe33000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:51.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.204+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe33c04a800 con 0x7fe3400f6820 2021-11-20T07:41:51.388 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.387+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe340027a00 con 0x7fe3400f6820 2021-11-20T07:41:51.388 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.387+0000 7fe336ffd700 1 -- 172.21.15.71:0/323474891 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7fe340027a00 con 0x7fe3400f6820 2021-11-20T07:41:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:51.389 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:51.390 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 msgr2=0x7fe32c05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:51.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 0x7fe32c05e7b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe3400f6680 tx=0x7fe33000b040).stop 2021-11-20T07:41:51.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 msgr2=0x7fe340106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:51.391 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe33c009f50 tx=0x7fe33c009b30).stop 2021-11-20T07:41:51.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 shutdown_connections 2021-11-20T07:41:51.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3400f5580 0x7fe340105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fe32c05c2f0 0x7fe32c05e7b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 --2- 172.21.15.71:0/323474891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3400f6820 0x7fe340106210 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:51.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 >> 172.21.15.71:0/323474891 conn(0x7fe3400f0ce0 msgr2=0x7fe3400fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:51.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 shutdown_connections 2021-11-20T07:41:51.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:51.389+0000 7fe3485dc700 1 -- 172.21.15.71:0/323474891 wait complete. 2021-11-20T07:41:51.395 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:51.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:51.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:51.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:51 smithi112 conmon[36062]: audit 2021-11-20T07:41:51.387561+0000 mon.smithi071 (mon. 2021-11-20T07:41:51.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:51 smithi112 conmon[36062]: 0) 785 : audit [DBG] from='client.? 172.21.15.71:0/323474891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:51 smithi071 conmon[33687]: audit 2021-11-20T07:41:51.387561+0000 mon.smithi071 (mon 2021-11-20T07:41:51.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:51 smithi071 conmon[33687]: .0) 785 : audit [DBG] from='client.? 172.21.15.71:0/323474891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:52.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:52.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- 172.21.15.71:0/835784961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f66b0 msgr2=0x7f74380f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:52.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 --2- 172.21.15.71:0/835784961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f66b0 0x7f74380f6ad0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7428004660 tx=0x7f7428009b30).stop 2021-11-20T07:41:52.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- 172.21.15.71:0/835784961 shutdown_connections 2021-11-20T07:41:52.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 --2- 172.21.15.71:0/835784961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f7950 0x7f74380f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:52.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 --2- 172.21.15.71:0/835784961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f66b0 0x7f74380f6ad0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:52.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- 172.21.15.71:0/835784961 >> 172.21.15.71:0/835784961 conn(0x7f74380f1dd0 msgr2=0x7f74380f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:52.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- 172.21.15.71:0/835784961 shutdown_connections 2021-11-20T07:41:52.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- 172.21.15.71:0/835784961 wait complete. 2021-11-20T07:41:52.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 Processor -- start 2021-11-20T07:41:52.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.530+0000 7f743e11f700 1 -- start start 2021-11-20T07:41:52.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f66b0 0x7f7438106f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:52.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:52.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438107a40 con 0x7f74380f7950 2021-11-20T07:41:52.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438107b80 con 0x7f74380f66b0 2021-11-20T07:41:52.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:52.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49144/0 (socket says 172.21.15.71:49144) 2021-11-20T07:41:52.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 -- 172.21.15.71:0/3120145631 learned_addr learned my addr 172.21.15.71:0/3120145631 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:52.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f74377fe700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f66b0 0x7f7438106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 -- 172.21.15.71:0/3120145631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f66b0 msgr2=0x7f7438106f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f66b0 0x7f7438106f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 -- 172.21.15.71:0/3120145631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7428005040 con 0x7f74380f7950 2021-11-20T07:41:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f742effd700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f742000b410 tx=0x7f7420006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:52.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74200125f0 con 0x7f74380f7950 2021-11-20T07:41:52.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7420012c00 con 0x7f74380f7950 2021-11-20T07:41:52.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743810c620 con 0x7f74380f7950 2021-11-20T07:41:52.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7420011d20 con 0x7f74380f7950 2021-11-20T07:41:52.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.531+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743810cbe0 con 0x7f74380f7950 2021-11-20T07:41:52.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.532+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438027a00 con 0x7f74380f7950 2021-11-20T07:41:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.533+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f742000f030 con 0x7f74380f7950 2021-11-20T07:41:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.533+0000 7f7434ff9700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 0x7f742405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.533+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f74200766b0 con 0x7f74380f7950 2021-11-20T07:41:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.533+0000 7f74377fe700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 0x7f742405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.534+0000 7f74377fe700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 0x7f742405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7428004630 tx=0x7f7428009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:52.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.537+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7420048f70 con 0x7f74380f7950 2021-11-20T07:41:52.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.720+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74380feec0 con 0x7f74380f7950 2021-11-20T07:41:52.723 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.721+0000 7f7434ff9700 1 -- 172.21.15.71:0/3120145631 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f742004c590 con 0x7f74380f7950 2021-11-20T07:41:52.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:52.724 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:52.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 msgr2=0x7f742405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:52.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 0x7f742405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7428004630 tx=0x7f7428009b30).stop 2021-11-20T07:41:52.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 msgr2=0x7f7438107400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:52.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f742000b410 tx=0x7f7420006ef0).stop 2021-11-20T07:41:52.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 shutdown_connections 2021-11-20T07:41:52.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74380f66b0 0x7f7438106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:52.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f742405c2a0 0x7f742405e760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:52.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 --2- 172.21.15.71:0/3120145631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74380f7950 0x7f7438107400 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:52.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 >> 172.21.15.71:0/3120145631 conn(0x7f74380f1dd0 msgr2=0x7f74380fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:52.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 shutdown_connections 2021-11-20T07:41:52.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:52.722+0000 7f743e11f700 1 -- 172.21.15.71:0/3120145631 wait complete. 2021-11-20T07:41:52.728 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:52.734 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:52 smithi071 conmon[33687]: cluster 2021-11-20T07:41:52.413068+0000 mgr.smithi071.ynqmwn (mgr.14162) 297 : cluster 2021-11-20T07:41:52.735 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:52 smithi071 conmon[33687]: [DBG] pgmap v270: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:41:52.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:52.736 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:52 smithi112 conmon[36062]: cluster 2021-11-20T07:41:52.413068+0000 mgr.smithi071.ynqmwn (mgr.14162) 297 : 2021-11-20T07:41:52.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:52 smithi112 conmon[36062]: cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-20T07:41:53.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:53.766 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:53 smithi071 conmon[33687]: audit 2021-11-20T07:41:52.722091+0000 mon.smithi071 (mon.0) 786 : audit 2021-11-20T07:41:53.767 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:53 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/3120145631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:53 smithi112 conmon[36062]: audit 2021-11-20T07:41:52.722091+0000 mon.smithi071 (mon. 2021-11-20T07:41:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:53 smithi112 conmon[36062]: 0) 786 : audit [DBG] from='client.? 172.21.15.71:0/3120145631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:53.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 -- 172.21.15.71:0/1794159073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 msgr2=0x7f10980f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:53.865 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 --2- 172.21.15.71:0/1794159073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f10980f6c80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f108c004660 tx=0x7f108c009b30).stop 2021-11-20T07:41:53.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 -- 172.21.15.71:0/1794159073 shutdown_connections 2021-11-20T07:41:53.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 --2- 172.21.15.71:0/1794159073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f10980f6c80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:53.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 --2- 172.21.15.71:0/1794159073 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 0x7f10980f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:53.866 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 -- 172.21.15.71:0/1794159073 >> 172.21.15.71:0/1794159073 conn(0x7f10980f0ce0 msgr2=0x7f10980f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:53.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 -- 172.21.15.71:0/1794159073 shutdown_connections 2021-11-20T07:41:53.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 -- 172.21.15.71:0/1794159073 wait complete. 2021-11-20T07:41:53.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.863+0000 7f109e02f700 1 Processor -- start 2021-11-20T07:41:53.867 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f109e02f700 1 -- start start 2021-11-20T07:41:53.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f109e02f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 0x7f1098060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:53.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f109e02f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:53.870 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f109e02f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109810c1a0 con 0x7f10980f6820 2021-11-20T07:41:53.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f109e02f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109810c2e0 con 0x7f10980f5580 2021-11-20T07:41:53.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:53.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f1096ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49154/0 (socket says 172.21.15.71:49154) 2021-11-20T07:41:53.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f1096ffd700 1 -- 172.21.15.71:0/3485063690 learned_addr learned my addr 172.21.15.71:0/3485063690 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:53.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.864+0000 7f10977fe700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 0x7f1098060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:53.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f1096ffd700 1 -- 172.21.15.71:0/3485063690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 msgr2=0x7f1098060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:53.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f1096ffd700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 0x7f1098060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:53.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f1096ffd700 1 -- 172.21.15.71:0/3485063690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c005040 con 0x7f10980f6820 2021-11-20T07:41:53.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f1096ffd700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f108c009830 tx=0x7f108c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:53.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c018070 con 0x7f10980f6820 2021-11-20T07:41:53.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f108c013410 con 0x7f10980f6820 2021-11-20T07:41:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109810c4e0 con 0x7f10980f6820 2021-11-20T07:41:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c01c3c0 con 0x7f10980f6820 2021-11-20T07:41:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.865+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f109810ca40 con 0x7f10980f6820 2021-11-20T07:41:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.866+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1098027a00 con 0x7f10980f6820 2021-11-20T07:41:53.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.866+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f108c01d420 con 0x7f10980f6820 2021-11-20T07:41:53.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.866+0000 7f107ffff700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 0x7f108005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:53.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.867+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f108c077f40 con 0x7f10980f6820 2021-11-20T07:41:53.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.867+0000 7f10977fe700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 0x7f108005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:53.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.867+0000 7f10977fe700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 0x7f108005e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f10980f6680 tx=0x7f108800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:53.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:53.870+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f108c0a42b0 con 0x7f10980f6820 2021-11-20T07:41:54.055 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.053+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10980fdd10 con 0x7f10980f6820 2021-11-20T07:41:54.056 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.054+0000 7f107ffff700 1 -- 172.21.15.71:0/3485063690 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f108c04a800 con 0x7f10980f6820 2021-11-20T07:41:54.056 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:54.056 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:39.421962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14650,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14650":{"gid":14650,"name":"cephfs.smithi071.xtyomb","rank":0,"incarnation":60,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6827/1534235240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1534235240},{"type":"v1","addr":"172.21.15.71:6827","nonce":1534235240}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:54.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 msgr2=0x7f108005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:54.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 0x7f108005e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f10980f6680 tx=0x7f108800b040).stop 2021-11-20T07:41:54.058 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 msgr2=0x7f109810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:54.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f108c009830 tx=0x7f108c009b30).stop 2021-11-20T07:41:54.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 shutdown_connections 2021-11-20T07:41:54.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10980f5580 0x7f1098060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:54.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f108005c2a0 0x7f108005e760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:54.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 --2- 172.21.15.71:0/3485063690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10980f6820 0x7f109810bcb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:54.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 >> 172.21.15.71:0/3485063690 conn(0x7f10980f0ce0 msgr2=0x7f10980fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:54.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 shutdown_connections 2021-11-20T07:41:54.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:54.055+0000 7f109e02f700 1 -- 172.21.15.71:0/3485063690 wait complete. 2021-11-20T07:41:54.060 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2021-11-20T07:41:54.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:39.421962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14650, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14650': {'gid': 14650, 'name': 'cephfs.smithi071.xtyomb', 'rank': 0, 'incarnation': 60, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6827/1534235240', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1534235240}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1534235240}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:54.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14650:up:clientreplay 2021-11-20T07:41:54.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: audit 2021-11-20T07:41:54.055139+0000 mon.smithi071 (mon.0) 787 : audit [DBG] 2021-11-20T07:41:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/3485063690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54.302556+0000 mon.smithi071 (mon.0) 788 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:41:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54.308976+0000 mon.smithi071 (mon.0) 789 : cluster 2021-11-20T07:41:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T07:41:54.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:41:54.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: 07:41:54.413446+0000 mgr.smithi071.ynqmwn (mgr.14162) 298 2021-11-20T07:41:54.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:54 smithi112 conmon[36062]: : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:41:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:54 smithi071 conmon[33687]: audit 2021-11-20T07:41:54.055139+0000 mon.smithi071 (mon.0) 787 : audit [DBG] from='client.? 172.21.15.71:0/3485063690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:54 smithi071 conmon[33687]: cluster 2021-11-20T07:41:54.302556+0000 mon.smithi071 (mon.0) 788 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb restarted 2021-11-20T07:41:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:54 smithi071 conmon[33687]: cluster 2021-11-20T07:41:54.308976+0000 mon.smithi071 (mon.0) 789 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T07:41:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:54 smithi071 conmon[33687]: cluster 2021-11-20T07:41:54.413446+0000 mgr.smithi071.ynqmwn (mgr.14162) 298 : cluster [DBG] 2021-11-20T07:41:54.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:54 smithi071 conmon[33687]: pgmap v272: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:41:55.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:55.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 -- 172.21.15.71:0/2557264589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f6680 msgr2=0x7f18840f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:55.199 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 --2- 172.21.15.71:0/2557264589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f6680 0x7f18840f6aa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f1874004660 tx=0x7f1874009b30).stop 2021-11-20T07:41:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 -- 172.21.15.71:0/2557264589 shutdown_connections 2021-11-20T07:41:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 --2- 172.21.15.71:0/2557264589 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f7920 0x7f18840f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 --2- 172.21.15.71:0/2557264589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f6680 0x7f18840f6aa0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 -- 172.21.15.71:0/2557264589 >> 172.21.15.71:0/2557264589 conn(0x7f18840f1de0 msgr2=0x7f18840f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:55.200 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 -- 172.21.15.71:0/2557264589 shutdown_connections 2021-11-20T07:41:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.195+0000 7f1889348700 1 -- 172.21.15.71:0/2557264589 wait complete. 2021-11-20T07:41:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 Processor -- start 2021-11-20T07:41:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 -- start start 2021-11-20T07:41:55.201 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f6680 0x7f1884106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:55.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:55.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884107990 con 0x7f18840f7920 2021-11-20T07:41:55.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1889348700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884107ad0 con 0x7f18840f6680 2021-11-20T07:41:55.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:55.202 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49176/0 (socket says 172.21.15.71:49176) 2021-11-20T07:41:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 -- 172.21.15.71:0/2953250421 learned_addr learned my addr 172.21.15.71:0/2953250421 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f1882ffd700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f6680 0x7f1884106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 -- 172.21.15.71:0/2953250421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f6680 msgr2=0x7f1884106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:55.203 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f6680 0x7f1884106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.196+0000 7f18827fc700 1 -- 172.21.15.71:0/2953250421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874005040 con 0x7f18840f7920 2021-11-20T07:41:55.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f18827fc700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f18780026d0 tx=0x7f1878009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:55.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1878009320 con 0x7f18840f7920 2021-11-20T07:41:55.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f187800f040 con 0x7f18840f7920 2021-11-20T07:41:55.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188410c570 con 0x7f18840f7920 2021-11-20T07:41:55.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18780148f0 con 0x7f18840f7920 2021-11-20T07:41:55.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.197+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188410cb30 con 0x7f18840f7920 2021-11-20T07:41:55.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.198+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1884027a00 con 0x7f18840f7920 2021-11-20T07:41:55.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.198+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f187801b060 con 0x7f18840f7920 2021-11-20T07:41:55.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.198+0000 7f186b7fe700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 0x7f186c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:55.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.198+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1878076250 con 0x7f18840f7920 2021-11-20T07:41:55.207 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.198+0000 7f1882ffd700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 0x7f186c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:55.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.199+0000 7f1882ffd700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 0x7f186c05e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1874009d60 tx=0x7f1874009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:55.208 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.202+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1878048b30 con 0x7f18840f7920 2021-11-20T07:41:55.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.392+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18840fee30 con 0x7f18840f7920 2021-11-20T07:41:55.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.392+0000 7f186b7fe700 1 -- 172.21.15.71:0/2953250421 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f187804c150 con 0x7f18840f7920 2021-11-20T07:41:55.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:55.395 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:54.477151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:replay","state_seq":3,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:55.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 msgr2=0x7f186c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:55.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 0x7f186c05e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1874009d60 tx=0x7f1874009b30).stop 2021-11-20T07:41:55.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 msgr2=0x7f18841073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:55.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f18780026d0 tx=0x7f1878009de0).stop 2021-11-20T07:41:55.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 shutdown_connections 2021-11-20T07:41:55.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18840f6680 0x7f1884106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f186c05c2a0 0x7f186c05e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 --2- 172.21.15.71:0/2953250421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18840f7920 0x7f18841073e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:55.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 >> 172.21.15.71:0/2953250421 conn(0x7f18840f1de0 msgr2=0x7f18840fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:55.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 shutdown_connections 2021-11-20T07:41:55.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:55.394+0000 7f1889348700 1 -- 172.21.15.71:0/2953250421 wait complete. 2021-11-20T07:41:55.399 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 69 2021-11-20T07:41:55.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:54.477151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:55.408 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:replay 2021-11-20T07:41:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54.477042+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: 790 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:boot 2021-11-20T07:41:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54.477112+0000 mon.smithi071 (mon.0) 791 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:41:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54.477841+0000 mon.smithi071 (mon.0) 792 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi112.bxcrgi=up:active} 3 up:standby, 1 failed 2021-11-20T07:41:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: audit 2021-11-20T07:41:54.477948+0000 2021-11-20T07:41:55.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: mon.smithi071 (mon.0) 793 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:41:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: cluster 2021-11-20T07:41:54. 2021-11-20T07:41:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: 482923+0000 mon.smithi071 (mon.0) 794 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: audit 2021-11-20 2021-11-20T07:41:55.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: T07:41:55.393722+0000 mon.smithi071 (mon. 2021-11-20T07:41:55.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:55 smithi112 conmon[36062]: 0) 795 : audit [DBG] from='client.? 172.21.15.71:0/2953250421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:55.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: cluster 2021-11-20T07:41:54.477042+0000 mon.smithi071 (mon.0) 790 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:boot 2021-11-20T07:41:55.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: cluster 2021-11-20T07:41:55.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: 2021-11-20T07:41:54.477112+0000 mon.smithi071 (mon.0) 791 : cluster [INF] Standby daemon mds.cephfs.smithi071.zaejme assigned to filesystem cephfs as rank 0 2021-11-20T07:41:55.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: cluster 2021-11-20T07:41:54.477841+0000 mon.smithi071 (mon.0 2021-11-20T07:41:55.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: ) 792 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi112.bxcrgi=up:active} 3 up:standby, 1 failed 2021-11-20T07:41:55.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:41:55.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: 07:41:54.477948+0000 mon.smithi071 ( 2021-11-20T07:41:55.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: mon.0) 793 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi071.xtyomb"}]: dispatch 2021-11-20T07:41:55.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:41:55.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: -20T07:41:54.482923+0000 mon.smithi071 (mon. 2021-11-20T07:41:55.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: 0) 794 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:55.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:41:55.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: -20T07:41:55.393722+0000 mon.smithi071 2021-11-20T07:41:55.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:55 smithi071 conmon[33687]: (mon.0) 795 : audit [DBG] from='client.? 172.21.15.71:0/2953250421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:56.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:56.535 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.530+0000 7f61031f6700 1 -- 172.21.15.71:0/413644225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 msgr2=0x7f60fc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:56.535 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.530+0000 7f61031f6700 1 --2- 172.21.15.71:0/413644225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc0f59a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f60f0004660 tx=0x7f60f0009b30).stop 2021-11-20T07:41:56.535 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- 172.21.15.71:0/413644225 shutdown_connections 2021-11-20T07:41:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 --2- 172.21.15.71:0/413644225 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 --2- 172.21.15.71:0/413644225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc0f59a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- 172.21.15.71:0/413644225 >> 172.21.15.71:0/413644225 conn(0x7f60fc0f0ce0 msgr2=0x7f60fc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:56.536 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- 172.21.15.71:0/413644225 shutdown_connections 2021-11-20T07:41:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- 172.21.15.71:0/413644225 wait complete. 2021-11-20T07:41:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 Processor -- start 2021-11-20T07:41:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- start start 2021-11-20T07:41:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:56.537 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc1051e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:56.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc105870 con 0x7f60fc0f5580 2021-11-20T07:41:56.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.531+0000 7f61031f6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc10a270 con 0x7f60fc0f6820 2021-11-20T07:41:56.538 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:56.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49194/0 (socket says 172.21.15.71:49194) 2021-11-20T07:41:56.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 -- 172.21.15.71:0/701723164 learned_addr learned my addr 172.21.15.71:0/701723164 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:56.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 -- 172.21.15.71:0/701723164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 msgr2=0x7f60fc1051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:56.539 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f60fbfff700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc1051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:56.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc1051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:56.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 -- 172.21.15.71:0/701723164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0005040 con 0x7f60fc0f5580 2021-11-20T07:41:56.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f60fbfff700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:41:56.540 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f6100f92700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f60f0009f50 tx=0x7f60f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:56.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60f0018070 con 0x7f60fc0f5580 2021-11-20T07:41:56.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc10a3b0 con 0x7f60fc0f5580 2021-11-20T07:41:56.541 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60f0013410 con 0x7f60fc0f5580 2021-11-20T07:41:56.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60f001c620 con 0x7f60fc0f5580 2021-11-20T07:41:56.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.532+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc10a8e0 con 0x7f60fc0f5580 2021-11-20T07:41:56.542 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.533+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc046330 con 0x7f60fc0f5580 2021-11-20T07:41:56.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.533+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f60f001d420 con 0x7f60fc0f5580 2021-11-20T07:41:56.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.534+0000 7f60f97fa700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 0x7f60ec060970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:56.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.534+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f60f0078200 con 0x7f60fc0f5580 2021-11-20T07:41:56.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.534+0000 7f60fbfff700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 0x7f60ec060970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:56.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.534+0000 7f60fbfff700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 0x7f60ec060970 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f60e800a7e0 tx=0x7f60e8008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:56.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.538+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60f004aa00 con 0x7f60fc0f5580 2021-11-20T07:41:56.726 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.724+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60fc0fcc80 con 0x7f60fc0f5580 2021-11-20T07:41:56.727 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.725+0000 7f60f97fa700 1 -- 172.21.15.71:0/701723164 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f60f004e020 con 0x7f60fc0f5580 2021-11-20T07:41:56.728 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:56.728 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:56.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.726+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 msgr2=0x7f60ec060970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:56.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.726+0000 7f61031f6700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 0x7f60ec060970 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f60e800a7e0 tx=0x7f60e8008040).stop 2021-11-20T07:41:56.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.726+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 msgr2=0x7f60fc104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:56.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.726+0000 7f61031f6700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f60f0009f50 tx=0x7f60f0009b30).stop 2021-11-20T07:41:56.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 shutdown_connections 2021-11-20T07:41:56.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f60ec05e4b0 0x7f60ec060970 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:56.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60fc0f5580 0x7f60fc104cf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:56.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 --2- 172.21.15.71:0/701723164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60fc0f6820 0x7f60fc1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:56.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 >> 172.21.15.71:0/701723164 conn(0x7f60fc0f0ce0 msgr2=0x7f60fc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:56.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 shutdown_connections 2021-11-20T07:41:56.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:56.727+0000 7f61031f6700 1 -- 172.21.15.71:0/701723164 wait complete. 2021-11-20T07:41:56.732 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 70 2021-11-20T07:41:56.739 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:56 smithi071 conmon[33687]: cluster 2021-11-20T07:41:55.488181+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:56.739 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:56 smithi071 conmon[33687]: 796 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:resolve 2021-11-20T07:41:56.740 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:56 smithi071 conmon[33687]: cluster 2021-11-20T07:41:55.488234+0000 mon.smithi071 (mon.0) 797 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:56.740 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:56 smithi071 conmon[33687]: cluster 2021-11-20T07:41:56.413848+0000 mgr.smithi071.ynqmwn (mgr.14162) 299 : cluster [DBG] pgmap v273: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T07:41:56.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:56.741 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:41:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:56 smithi112 conmon[36062]: cluster 2021-11-20T07:41:55.488181+0000 mon.smithi071 (mon.0) 796 : cluster [DBG] 2021-11-20T07:41:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:56 smithi112 conmon[36062]: mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:resolve 2021-11-20T07:41:56.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:56 smithi112 conmon[36062]: cluster 2021-11-20T07:41:55.488234+0000 mon.smithi071 (mon.0) 797 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:56.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:56 smithi112 conmon[36062]: cluster 2021-11-20T07:41:56.413848+0000 mgr.smithi071.ynqmwn (mgr.14162) 299 : cluster [DBG] pgmap v273: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T07:41:57.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:57.770 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:57 smithi071 conmon[33687]: audit 2021-11-20T07:41:56.726248+0000 mon.smithi071 (mon.0) 798 : audit [DBG] from='client.? 172.21.15.71:0/701723164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:57.771 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:57 smithi071 conmon[33687]: 2021-11-20T07:41:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:57 smithi112 conmon[36062]: audit 2021-11-20T07:41:56.726248+0000 mon.smithi071 (mon.0) 2021-11-20T07:41:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:57 smithi112 conmon[36062]: 798 : audit [DBG] from='client.? 172.21.15.71:0/701723164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:57.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 -- 172.21.15.71:0/1147515500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f63f0 msgr2=0x7f47d80f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:57.871 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 --2- 172.21.15.71:0/1147515500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f63f0 0x7f47d80f6850 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f47c8004660 tx=0x7f47c8009b30).stop 2021-11-20T07:41:57.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 -- 172.21.15.71:0/1147515500 shutdown_connections 2021-11-20T07:41:57.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 --2- 172.21.15.71:0/1147515500 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f63f0 0x7f47d80f6850 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:57.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 --2- 172.21.15.71:0/1147515500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f5a50 0x7f47d80f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:57.872 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 -- 172.21.15.71:0/1147515500 >> 172.21.15.71:0/1147515500 conn(0x7f47d80f0360 msgr2=0x7f47d80f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:57.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 -- 172.21.15.71:0/1147515500 shutdown_connections 2021-11-20T07:41:57.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.866+0000 7f47dc972700 1 -- 172.21.15.71:0/1147515500 wait complete. 2021-11-20T07:41:57.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 Processor -- start 2021-11-20T07:41:57.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 -- start start 2021-11-20T07:41:57.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:57.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f63f0 0x7f47d810e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:57.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80f5430 con 0x7f47d80f5a50 2021-11-20T07:41:57.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47dc972700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80f5570 con 0x7f47d80f63f0 2021-11-20T07:41:57.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47d77fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:57.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47d77fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49208/0 (socket says 172.21.15.71:49208) 2021-11-20T07:41:57.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.867+0000 7f47d77fe700 1 -- 172.21.15.71:0/1267195770 learned_addr learned my addr 172.21.15.71:0/1267195770 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:57.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47d6ffd700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f63f0 0x7f47d810e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:57.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47d77fe700 1 -- 172.21.15.71:0/1267195770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f63f0 msgr2=0x7f47d810e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:57.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47d77fe700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f63f0 0x7f47d810e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:57.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47d77fe700 1 -- 172.21.15.71:0/1267195770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8005040 con 0x7f47d80f5a50 2021-11-20T07:41:57.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47d77fe700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f47cc00c530 tx=0x7f47cc007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:57.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47cc00f650 con 0x7f47d80f5a50 2021-11-20T07:41:57.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47cc00fc60 con 0x7f47d80f5a50 2021-11-20T07:41:57.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d810e810 con 0x7f47d80f5a50 2021-11-20T07:41:57.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47cc012c60 con 0x7f47d80f5a50 2021-11-20T07:41:57.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.868+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d810ec90 con 0x7f47d80f5a50 2021-11-20T07:41:57.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.869+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d8045ad0 con 0x7f47d80f5a50 2021-11-20T07:41:57.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.869+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f47cc019030 con 0x7f47d80f5a50 2021-11-20T07:41:57.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.870+0000 7f47bffff700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 0x7f47c005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:57.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.870+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47cc075e90 con 0x7f47d80f5a50 2021-11-20T07:41:57.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.870+0000 7f47d6ffd700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 0x7f47c005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:57.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.870+0000 7f47d6ffd700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 0x7f47c005e7c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f47c8004630 tx=0x7f47c8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:57.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:57.873+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47cc0462b0 con 0x7f47d80f5a50 2021-11-20T07:41:58.059 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.058+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d8027a00 con 0x7f47d80f5a50 2021-11-20T07:41:58.060 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.058+0000 7f47bffff700 1 -- 172.21.15.71:0/1267195770 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f47cc017090 con 0x7f47d80f5a50 2021-11-20T07:41:58.060 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:58.061 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 msgr2=0x7f47c005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 0x7f47c005e7c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f47c8004630 tx=0x7f47c8009b30).stop 2021-11-20T07:41:58.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 msgr2=0x7f47d810be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f47cc00c530 tx=0x7f47cc007e10).stop 2021-11-20T07:41:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 shutdown_connections 2021-11-20T07:41:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47c005c300 0x7f47c005e7c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47d80f5a50 0x7f47d810be20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:58.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 --2- 172.21.15.71:0/1267195770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47d80f63f0 0x7f47d810e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 >> 172.21.15.71:0/1267195770 conn(0x7f47d80f0360 msgr2=0x7f47d80fa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 shutdown_connections 2021-11-20T07:41:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:58.060+0000 7f47dc972700 1 -- 172.21.15.71:0/1267195770 wait complete. 2021-11-20T07:41:58.064 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 70 2021-11-20T07:41:58.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:58.074 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:41:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:58 smithi112 conmon[36062]: audit 2021-11-20T07:41:58.059524+0000 mon.smithi071 (mon.0) 799 : audit [DBG] 2021-11-20T07:41:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:58 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/1267195770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:58 smithi112 conmon[36062]: cluster 2021-11-20T07:41:58.414272+0000 mgr.smithi071.ynqmwn (mgr.14162) 300 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:41:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:58 smithi071 conmon[33687]: audit 2021-11-20T07:41:58.059524+0000 mon.smithi071 (mon.0) 799 : audit 2021-11-20T07:41:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:58 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/1267195770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:58 smithi071 conmon[33687]: cluster 2021-11-20T07:41:58.414272+0000 mgr.smithi071.ynqmwn (mgr.14162) 300 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T07:41:59.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:41:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 -- 172.21.15.71:0/3373490648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06b960 msgr2=0x7f69ac069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 --2- 172.21.15.71:0/3373490648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06b960 0x7f69ac069e60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f69a8004660 tx=0x7f69a8009b30).stop 2021-11-20T07:41:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 -- 172.21.15.71:0/3373490648 shutdown_connections 2021-11-20T07:41:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 --2- 172.21.15.71:0/3373490648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06a350 0x7f69ac06a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:41:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 --2- 172.21.15.71:0/3373490648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06b960 0x7f69ac069e60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 -- 172.21.15.71:0/3373490648 >> 172.21.15.71:0/3373490648 conn(0x7f69ac0f1d10 msgr2=0x7f69ac0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 -- 172.21.15.71:0/3373490648 shutdown_connections 2021-11-20T07:41:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.201+0000 7f69b3018700 1 -- 172.21.15.71:0/3373490648 wait complete. 2021-11-20T07:41:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 Processor -- start 2021-11-20T07:41:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 -- start start 2021-11-20T07:41:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06b960 0x7f69ac109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac109a40 con 0x7f69ac06a350 2021-11-20T07:41:59.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b3018700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac10a2f0 con 0x7f69ac06b960 2021-11-20T07:41:59.210 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.202+0000 7f69b2016700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:59.211 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49222/0 (socket says 172.21.15.71:49222) 2021-11-20T07:41:59.211 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 -- 172.21.15.71:0/1486730591 learned_addr learned my addr 172.21.15.71:0/1486730591 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:41:59.211 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b1815700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06b960 0x7f69ac109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:59.211 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 -- 172.21.15.71:0/1486730591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06b960 msgr2=0x7f69ac109400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:59.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06b960 0x7f69ac109400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:59.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 -- 172.21.15.71:0/1486730591 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8005040 con 0x7f69ac06a350 2021-11-20T07:41:59.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b2016700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f69a8009830 tx=0x7f69a8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:59.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a8018070 con 0x7f69ac06a350 2021-11-20T07:41:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69a8013410 con 0x7f69ac06a350 2021-11-20T07:41:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac10a570 con 0x7f69ac06a350 2021-11-20T07:41:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a801c620 con 0x7f69ac06a350 2021-11-20T07:41:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.203+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac10aad0 con 0x7f69ac06a350 2021-11-20T07:41:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.204+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0f59c0 con 0x7f69ac06a350 2021-11-20T07:41:59.214 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.205+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f69a801c8e0 con 0x7f69ac06a350 2021-11-20T07:41:59.214 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.206+0000 7f699effd700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 0x7f699805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:41:59.214 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.206+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f69a8078570 con 0x7f69ac06a350 2021-11-20T07:41:59.214 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.206+0000 7f69b1815700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 0x7f699805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:41:59.215 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.207+0000 7f69b1815700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 0x7f699805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f69a000b790 tx=0x7f69a0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:41:59.215 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.211+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69a804ae60 con 0x7f69ac06a350 2021-11-20T07:41:59.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.396+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69ac045ad0 con 0x7f69ac06a350 2021-11-20T07:41:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.397+0000 7f699effd700 1 -- 172.21.15.71:0/1486730591 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f69a804e480 con 0x7f69ac06a350 2021-11-20T07:41:59.399 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:41:59.400 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:41:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 msgr2=0x7f699805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 0x7f699805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f69a000b790 tx=0x7f69a0008040).stop 2021-11-20T07:41:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 msgr2=0x7f69ac108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:41:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f69a8009830 tx=0x7f69a8009b30).stop 2021-11-20T07:41:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 shutdown_connections 2021-11-20T07:41:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f699805c300 0x7f699805e7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69ac06a350 0x7f69ac108f10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 --2- 172.21.15.71:0/1486730591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69ac06b960 0x7f69ac109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:41:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.398+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 >> 172.21.15.71:0/1486730591 conn(0x7f69ac0f1d10 msgr2=0x7f69ac0fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:41:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.399+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 shutdown_connections 2021-11-20T07:41:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:41:59.399+0000 7f69b3018700 1 -- 172.21.15.71:0/1486730591 wait complete. 2021-11-20T07:41:59.404 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:41:59.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:41:59.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:41:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:59 smithi112 conmon[36062]: cluster 2021-11-20T07:41:58.499003+0000 mon.smithi071 ( 2021-11-20T07:41:59.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:59 smithi112 conmon[36062]: mon.0) 800 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:standby 2021-11-20T07:41:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:59 smithi112 conmon[36062]: cluster 2021-11-20T07:41:58.499072+0000 mon.smithi071 (mon.0) 801 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:59.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:41:59 smithi112 conmon[36062]: audit 2021-11-20T07:41:59.398307+0000 mon.smithi071 (mon.0) 802 : audit [DBG] from='client.? 172.21.15.71:0/1486730591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:41:59.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:59 smithi071 conmon[33687]: cluster 2021-11-20T07:41:58.499003+0000 mon.smithi071 (mon.0) 800 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:standby 2021-11-20T07:41:59.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:59 smithi071 conmon[33687]: cluster 2021-11-20T07:41:58.499072+0000 mon.smithi071 (mon.0) 801 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2 up:standby 2021-11-20T07:41:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:59 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:41:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:59 smithi071 conmon[33687]: -20T07:41:59.398307+0000 mon.smithi071 (mon.0) 802 : audit [DBG] 2021-11-20T07:41:59.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:41:59 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/1486730591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:00.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:00.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- 172.21.15.71:0/3154923895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 msgr2=0x7ff6200f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:00.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 --2- 172.21.15.71:0/3154923895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6200f7a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff608004660 tx=0x7ff608009b30).stop 2021-11-20T07:42:00.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- 172.21.15.71:0/3154923895 shutdown_connections 2021-11-20T07:42:00.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 --2- 172.21.15.71:0/3154923895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6200f7a80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.545 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 --2- 172.21.15.71:0/3154923895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff6200f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- 172.21.15.71:0/3154923895 >> 172.21.15.71:0/3154923895 conn(0x7ff6200f0440 msgr2=0x7ff6200f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:00.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- 172.21.15.71:0/3154923895 shutdown_connections 2021-11-20T07:42:00.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- 172.21.15.71:0/3154923895 wait complete. 2021-11-20T07:42:00.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 Processor -- start 2021-11-20T07:42:00.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- start start 2021-11-20T07:42:00.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff620106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:00.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:00.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff620107970 con 0x7ff6200f5300 2021-11-20T07:42:00.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.535+0000 7ff625db9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff620107ab0 con 0x7ff6200f4960 2021-11-20T07:42:00.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:00.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49234/0 (socket says 172.21.15.71:49234) 2021-11-20T07:42:00.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 -- 172.21.15.71:0/3444155047 learned_addr learned my addr 172.21.15.71:0/3444155047 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:00.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff61f7fe700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff620106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:00.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 -- 172.21.15.71:0/3444155047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 msgr2=0x7ff620106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:00.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff620106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 -- 172.21.15.71:0/3444155047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff608005040 con 0x7ff6200f5300 2021-11-20T07:42:00.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff61f7fe700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff620106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:42:00.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff617fff700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff608004e30 tx=0x7ff608009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:00.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff608018070 con 0x7ff6200f5300 2021-11-20T07:42:00.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff608013410 con 0x7ff6200f5300 2021-11-20T07:42:00.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff60801c3c0 con 0x7ff6200f5300 2021-11-20T07:42:00.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62010c4f0 con 0x7ff6200f5300 2021-11-20T07:42:00.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.536+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62010cab0 con 0x7ff6200f5300 2021-11-20T07:42:00.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.537+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff620027a00 con 0x7ff6200f5300 2021-11-20T07:42:00.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.537+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff60801d420 con 0x7ff6200f5300 2021-11-20T07:42:00.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.538+0000 7ff61cff9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:00.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.538+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff608077f20 con 0x7ff6200f5300 2021-11-20T07:42:00.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.538+0000 7ff61f7fe700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:00.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.538+0000 7ff61f7fe700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 0x7ff60c05e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff62005f7e0 tx=0x7ff61000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:00.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.542+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6080a42b0 con 0x7ff6200f5300 2021-11-20T07:42:00.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.727+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6200fee50 con 0x7ff6200f5300 2021-11-20T07:42:00.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.727+0000 7ff61cff9700 1 -- 172.21.15.71:0/3444155047 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7ff60804a780 con 0x7ff6200f5300 2021-11-20T07:42:00.730 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:00.730 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:00.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 msgr2=0x7ff60c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:00.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 0x7ff60c05e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff62005f7e0 tx=0x7ff61000b040).stop 2021-11-20T07:42:00.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 msgr2=0x7ff6201073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:00.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff608004e30 tx=0x7ff608009b30).stop 2021-11-20T07:42:00.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 shutdown_connections 2021-11-20T07:42:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200f4960 0x7ff620106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 --2- 172.21.15.71:0/3444155047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6200f5300 0x7ff6201073c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 >> 172.21.15.71:0/3444155047 conn(0x7ff6200f0440 msgr2=0x7ff6200fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 shutdown_connections 2021-11-20T07:42:00.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:00.729+0000 7ff625db9700 1 -- 172.21.15.71:0/3444155047 wait complete. 2021-11-20T07:42:00.734 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:42:00.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:00.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:01.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:01.772 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:01 smithi071 conmon[33687]: cluster 2021-11-20T07:42:00.414820+0000 mgr.smithi071.ynqmwn (mgr.14162) 301 : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:01.773 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:01 smithi071 conmon[33687]: audit 2021-11-20 2021-11-20T07:42:01.773 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:01 smithi071 conmon[33687]: T07:42:00.728768+0000 mon.smithi071 (mon.0) 803 : audit [DBG] from='client.? 172.21.15.71:0/3444155047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:01 smithi112 conmon[36062]: cluster 2021-11-20T07:42:00.414820+0000 mgr.smithi071.ynqmwn (mgr.14162) 301 : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:01.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:01 smithi112 conmon[36062]: audit 2021-11-20T07:42:00.728768+0000 mon.smithi071 (mon.0) 803 : audit [DBG] from='client.? 172.21.15.71:0/3444155047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:01.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- 172.21.15.71:0/838556372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f7940 msgr2=0x7f7f180f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:01.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/838556372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f7940 0x7f7f180f7da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f7f0c0099e0 tx=0x7f7f0c004d10).stop 2021-11-20T07:42:01.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- 172.21.15.71:0/838556372 shutdown_connections 2021-11-20T07:42:01.873 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/838556372 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f7940 0x7f7f180f7da0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:01.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/838556372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f66a0 0x7f7f180f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:01.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- 172.21.15.71:0/838556372 >> 172.21.15.71:0/838556372 conn(0x7f7f180f1de0 msgr2=0x7f7f180f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:01.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- 172.21.15.71:0/838556372 shutdown_connections 2021-11-20T07:42:01.874 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- 172.21.15.71:0/838556372 wait complete. 2021-11-20T07:42:01.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 Processor -- start 2021-11-20T07:42:01.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.869+0000 7f7f1eee9700 1 -- start start 2021-11-20T07:42:01.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1eee9700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:01.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1eee9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f7940 0x7f7f18107500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:01.875 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1eee9700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18107b40 con 0x7f7f180f66a0 2021-11-20T07:42:01.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1eee9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18107c80 con 0x7f7f180f7940 2021-11-20T07:42:01.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:01.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49244/0 (socket says 172.21.15.71:49244) 2021-11-20T07:42:01.876 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 -- 172.21.15.71:0/3563269843 learned_addr learned my addr 172.21.15.71:0/3563269843 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:01.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f17fff700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f7940 0x7f7f18107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:01.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 -- 172.21.15.71:0/3563269843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f7940 msgr2=0x7f7f18107500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:01.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f7940 0x7f7f18107500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:01.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 -- 172.21.15.71:0/3563269843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c009640 con 0x7f7f180f66a0 2021-11-20T07:42:01.877 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1cc85700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7f080093f0 tx=0x7f7f08006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:01.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f08013640 con 0x7f7f180f66a0 2021-11-20T07:42:01.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f08013c50 con 0x7f7f180f66a0 2021-11-20T07:42:01.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.870+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1810c720 con 0x7f7f180f66a0 2021-11-20T07:42:01.878 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.871+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f08012d10 con 0x7f7f180f66a0 2021-11-20T07:42:01.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.871+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1810cc60 con 0x7f7f180f66a0 2021-11-20T07:42:01.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.872+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18027a00 con 0x7f7f180f66a0 2021-11-20T07:42:01.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.872+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7f0801a060 con 0x7f7f180f66a0 2021-11-20T07:42:01.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.872+0000 7f7f157fa700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 0x7f7f0005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:01.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.872+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f08076310 con 0x7f7f180f66a0 2021-11-20T07:42:01.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.872+0000 7f7f17fff700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 0x7f7f0005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:01.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.873+0000 7f7f17fff700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 0x7f7f0005e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7f0c003380 tx=0x7f7f0c004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:01.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:01.876+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f08048b70 con 0x7f7f180f66a0 2021-11-20T07:42:02.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.060+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f180feed0 con 0x7f7f180f66a0 2021-11-20T07:42:02.212 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.061+0000 7f7f157fa700 1 -- 172.21.15.71:0/3563269843 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f7f0804c190 con 0x7f7f180f66a0 2021-11-20T07:42:02.213 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:02.213 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:02.215 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 msgr2=0x7f7f0005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:02.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 0x7f7f0005e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7f0c003380 tx=0x7f7f0c004d10).stop 2021-11-20T07:42:02.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 msgr2=0x7f7f18107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:02.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7f080093f0 tx=0x7f7f08006ec0).stop 2021-11-20T07:42:02.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 shutdown_connections 2021-11-20T07:42:02.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f7f0005c2a0 0x7f7f0005e760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:02.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7f180f66a0 0x7f7f18107010 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:02.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 --2- 172.21.15.71:0/3563269843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f180f7940 0x7f7f18107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:02.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 >> 172.21.15.71:0/3563269843 conn(0x7f7f180f1de0 msgr2=0x7f7f180fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:02.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 shutdown_connections 2021-11-20T07:42:02.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:02.062+0000 7f7f1eee9700 1 -- 172.21.15.71:0/3563269843 wait complete. 2021-11-20T07:42:02.219 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:42:02.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:02.220 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:02.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:02 smithi112 conmon[36062]: audit 2021-11-20T07:42:02.062128+0000 mon.smithi071 (mon. 2021-11-20T07:42:02.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:02 smithi112 conmon[36062]: 0) 804 : audit [DBG] from='client.? 172.21.15.71:0/3563269843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:02.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:02 smithi071 conmon[33687]: audit 2021-11-20T07:42:02.062128+0000 mon.smithi071 (mon.0) 804 : audit [DBG] from='client.? 172.21.15.71:0/3563269843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:03.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.350+0000 7fd30c6c6700 1 -- 172.21.15.71:0/2347662867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 msgr2=0x7fd3040f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.350+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/2347662867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3040f6cb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd2f4004660 tx=0x7fd2f4009b30).stop 2021-11-20T07:42:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- 172.21.15.71:0/2347662867 shutdown_connections 2021-11-20T07:42:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/2347662867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3040f6cb0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/2347662867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 0x7fd3040f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- 172.21.15.71:0/2347662867 >> 172.21.15.71:0/2347662867 conn(0x7fd3040f0d10 msgr2=0x7fd3040f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- 172.21.15.71:0/2347662867 shutdown_connections 2021-11-20T07:42:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- 172.21.15.71:0/2347662867 wait complete. 2021-11-20T07:42:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 Processor -- start 2021-11-20T07:42:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- start start 2021-11-20T07:42:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 0x7fd304105de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304106910 con 0x7fd3040f6850 2021-11-20T07:42:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd30c6c6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304106a50 con 0x7fd3040f55b0 2021-11-20T07:42:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd309c61700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd309c61700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49258/0 (socket says 172.21.15.71:49258) 2021-11-20T07:42:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.351+0000 7fd309c61700 1 -- 172.21.15.71:0/3202367363 learned_addr learned my addr 172.21.15.71:0/3202367363 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd309c61700 1 -- 172.21.15.71:0/3202367363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 msgr2=0x7fd304105de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd30a462700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 0x7fd304105de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd309c61700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 0x7fd304105de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd309c61700 1 -- 172.21.15.71:0/3202367363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4005040 con 0x7fd3040f6850 2021-11-20T07:42:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd309c61700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd2f4007a40 tx=0x7fd2f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f4018070 con 0x7fd3040f6850 2021-11-20T07:42:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2f40097d0 con 0x7fd3040f6850 2021-11-20T07:42:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30410b490 con 0x7fd3040f6850 2021-11-20T07:42:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f401c620 con 0x7fd3040f6850 2021-11-20T07:42:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.352+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30410b9f0 con 0x7fd3040f6850 2021-11-20T07:42:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.353+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd304027a00 con 0x7fd3040f6850 2021-11-20T07:42:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.353+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd2f401c890 con 0x7fd3040f6850 2021-11-20T07:42:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.354+0000 7fd2faffd700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 0x7fd2f005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.354+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2f40782b0 con 0x7fd3040f6850 2021-11-20T07:42:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.354+0000 7fd30a462700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 0x7fd2f005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.354+0000 7fd30a462700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 0x7fd2f005e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd3040f66b0 tx=0x7fd300008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.357+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2f404ab10 con 0x7fd3040f6850 2021-11-20T07:42:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.541+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3040fdde0 con 0x7fd3040f6850 2021-11-20T07:42:03.544 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.542+0000 7fd2faffd700 1 -- 172.21.15.71:0/3202367363 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fd2f404e130 con 0x7fd3040f6850 2021-11-20T07:42:03.544 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:03.545 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 msgr2=0x7fd2f005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 0x7fd2f005e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd3040f66b0 tx=0x7fd300008040).stop 2021-11-20T07:42:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 msgr2=0x7fd3041062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd2f4007a40 tx=0x7fd2f4009b30).stop 2021-11-20T07:42:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 shutdown_connections 2021-11-20T07:42:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd3040f55b0 0x7fd304105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd2f005c2a0 0x7fd2f005e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 --2- 172.21.15.71:0/3202367363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3040f6850 0x7fd3041062d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.543+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 >> 172.21.15.71:0/3202367363 conn(0x7fd3040f0d10 msgr2=0x7fd3040fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.544+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 shutdown_connections 2021-11-20T07:42:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:03.544+0000 7fd30c6c6700 1 -- 172.21.15.71:0/3202367363 wait complete. 2021-11-20T07:42:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:42:03.556 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:03 smithi071 conmon[33687]: cluster 2021-11-20T07:42:02.415241+0000 mgr.smithi071.ynqmwn (mgr.14162) 302 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:03.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:03.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:03.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:03 smithi112 conmon[36062]: cluster 2021-11-20T07:42:02.415241+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:42:03.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:03 smithi112 conmon[36062]: 302 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:04.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:04.690 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 -- 172.21.15.71:0/1064172110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f6bd0 msgr2=0x7f83440f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 --2- 172.21.15.71:0/1064172110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f6bd0 0x7f83440f6ff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f8340004660 tx=0x7f8340009b30).stop 2021-11-20T07:42:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 -- 172.21.15.71:0/1064172110 shutdown_connections 2021-11-20T07:42:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 --2- 172.21.15.71:0/1064172110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83440f7570 0x7f83440f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 --2- 172.21.15.71:0/1064172110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f6bd0 0x7f83440f6ff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 -- 172.21.15.71:0/1064172110 >> 172.21.15.71:0/1064172110 conn(0x7f83440f04a0 msgr2=0x7f83440f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 -- 172.21.15.71:0/1064172110 shutdown_connections 2021-11-20T07:42:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.686+0000 7f834b979700 1 -- 172.21.15.71:0/1064172110 wait complete. 2021-11-20T07:42:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 Processor -- start 2021-11-20T07:42:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 -- start start 2021-11-20T07:42:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83440f6bd0 0x7f83440f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834410c1a0 con 0x7f83440f7570 2021-11-20T07:42:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f834b979700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834410c2e0 con 0x7f83440f6bd0 2021-11-20T07:42:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49268/0 (socket says 172.21.15.71:49268) 2021-11-20T07:42:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 -- 172.21.15.71:0/1055413308 learned_addr learned my addr 172.21.15.71:0/1055413308 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 -- 172.21.15.71:0/1055413308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83440f6bd0 msgr2=0x7f83440f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83440f6bd0 0x7f83440f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 -- 172.21.15.71:0/1055413308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340005040 con 0x7f83440f7570 2021-11-20T07:42:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.687+0000 7f8348f14700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f833400b410 tx=0x7f8334006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.688+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83340125f0 con 0x7f83440f7570 2021-11-20T07:42:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.688+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8334012c00 con 0x7f83440f7570 2021-11-20T07:42:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.688+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834410c5c0 con 0x7f83440f7570 2021-11-20T07:42:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.688+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8334002aa0 con 0x7f83440f7570 2021-11-20T07:42:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.688+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834410cb80 con 0x7f83440f7570 2021-11-20T07:42:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.689+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8344027a00 con 0x7f83440f7570 2021-11-20T07:42:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.689+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f833400f030 con 0x7f83440f7570 2021-11-20T07:42:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.689+0000 7f8339ffb700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 0x7f833005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.689+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8334076820 con 0x7f83440f7570 2021-11-20T07:42:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.689+0000 7f8349715700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 0x7f833005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.690+0000 7f8349715700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 0x7f833005e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8340000c00 tx=0x7f8340009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.693+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8334049080 con 0x7f83440f7570 2021-11-20T07:42:04.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:04 smithi112 conmon[36062]: audit 2021-11-20T07:42:03. 2021-11-20T07:42:04.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:04 smithi112 conmon[36062]: 543265+0000 mon.smithi071 (mon.0) 805 : audit [DBG] from='client.? 172.21.15.71:0/3202367363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:04.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:04 smithi071 conmon[33687]: audit 2021-11-20T07:42:03.543265+0000 mon.smithi071 (mon.0) 805 : audit [DBG] from='client.? 172.21.15.71:0/3202367363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:04.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.881+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83440fdde0 con 0x7f83440f7570 2021-11-20T07:42:04.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.881+0000 7f8339ffb700 1 -- 172.21.15.71:0/1055413308 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f833404c6a0 con 0x7f83440f7570 2021-11-20T07:42:04.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:04.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:04.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 msgr2=0x7f833005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:04.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 0x7f833005e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8340000c00 tx=0x7f8340009b30).stop 2021-11-20T07:42:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 msgr2=0x7f834410bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f833400b410 tx=0x7f8334006ef0).stop 2021-11-20T07:42:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 shutdown_connections 2021-11-20T07:42:04.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83440f6bd0 0x7f83440f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f833005c2a0 0x7f833005e760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 --2- 172.21.15.71:0/1055413308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f83440f7570 0x7f834410bcb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 >> 172.21.15.71:0/1055413308 conn(0x7f83440f04a0 msgr2=0x7f83440fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 shutdown_connections 2021-11-20T07:42:04.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:04.883+0000 7f834b979700 1 -- 172.21.15.71:0/1055413308 wait complete. 2021-11-20T07:42:04.888 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:42:04.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:04.897 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:05 smithi112 conmon[36062]: cluster 2021-11-20T07:42:04.415761+0000 mgr.smithi071.ynqmwn (mgr.14162) 303 2021-11-20T07:42:05.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:05 smithi112 conmon[36062]: : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:05.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:05 smithi112 conmon[36062]: audit 2021-11-20T07:42:04.882459+0000 mon.smithi071 (mon.0) 806 : audit [DBG] from='client.? 172.21.15.71:0/1055413308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:05 smithi071 conmon[33687]: cluster 2021-11-20T07:42:04.415761+0000 mgr.smithi071.ynqmwn (mgr.14162) 303 2021-11-20T07:42:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:05 smithi071 conmon[33687]: : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:05.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:05 smithi071 conmon[33687]: audit 2021-11-20T07:42:04.882459+0000 mon.smithi071 (mon.0) 806 : audit [DBG] from='client.? 172.21.15.71:0/1055413308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:05.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:06.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 -- 172.21.15.71:0/1706976812 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 msgr2=0x7f9c440f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:06.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/1706976812 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c440f5970 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f9c34004660 tx=0x7f9c34009b30).stop 2021-11-20T07:42:06.027 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 -- 172.21.15.71:0/1706976812 shutdown_connections 2021-11-20T07:42:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/1706976812 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c440f5970 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/1706976812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 -- 172.21.15.71:0/1706976812 >> 172.21.15.71:0/1706976812 conn(0x7f9c440f0440 msgr2=0x7f9c440f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 -- 172.21.15.71:0/1706976812 shutdown_connections 2021-11-20T07:42:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 -- 172.21.15.71:0/1706976812 wait complete. 2021-11-20T07:42:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.022+0000 7f9c4ba80700 1 Processor -- start 2021-11-20T07:42:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 -- start start 2021-11-20T07:42:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410c1c0 con 0x7f9c440f7520 2021-11-20T07:42:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410c300 con 0x7f9c440f6b80 2021-11-20T07:42:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49278/0 (socket says 172.21.15.71:49278) 2021-11-20T07:42:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 -- 172.21.15.71:0/2141321279 learned_addr learned my addr 172.21.15.71:0/2141321279 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4981c700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 -- 172.21.15.71:0/2141321279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 msgr2=0x7f9c440f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 -- 172.21.15.71:0/2141321279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34005040 con 0x7f9c440f7520 2021-11-20T07:42:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4981c700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4901b700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9c34009f50 tx=0x7f9c34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c34018070 con 0x7f9c440f7520 2021-11-20T07:42:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c34013410 con 0x7f9c440f7520 2021-11-20T07:42:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4410c500 con 0x7f9c440f7520 2021-11-20T07:42:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3401c620 con 0x7f9c440f7520 2021-11-20T07:42:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.023+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4410ca30 con 0x7f9c440f7520 2021-11-20T07:42:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.024+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c44046330 con 0x7f9c440f7520 2021-11-20T07:42:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.025+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9c3401d420 con 0x7f9c440f7520 2021-11-20T07:42:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.025+0000 7f9c3a7fc700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 0x7f9c3005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.025+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c340780f0 con 0x7f9c440f7520 2021-11-20T07:42:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.025+0000 7f9c4981c700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 0x7f9c3005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.025+0000 7f9c4981c700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 0x7f9c3005e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9c4405f7a0 tx=0x7f9c40009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:06.035 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.029+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c3404a9d0 con 0x7f9c440f7520 2021-11-20T07:42:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.213+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c44027a00 con 0x7f9c440f7520 2021-11-20T07:42:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.214+0000 7f9c3a7fc700 1 -- 172.21.15.71:0/2141321279 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f9c3404dff0 con 0x7f9c440f7520 2021-11-20T07:42:06.217 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:06.217 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71},{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:41:55.480859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":24373},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":63,"state":"up:active","state_seq":20,"addr":"172.21.15.112:6825/2992700860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2992700860},{"type":"v1","addr":"172.21.15.112:6825","nonce":2992700860}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:06.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 msgr2=0x7f9c3005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 0x7f9c3005e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9c4405f7a0 tx=0x7f9c40009040).stop 2021-11-20T07:42:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 msgr2=0x7f9c4410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9c34009f50 tx=0x7f9c34009b30).stop 2021-11-20T07:42:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 shutdown_connections 2021-11-20T07:42:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c440f6b80 0x7f9c440f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9c3005c2a0 0x7f9c3005e760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 --2- 172.21.15.71:0/2141321279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c440f7520 0x7f9c4410bcd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 >> 172.21.15.71:0/2141321279 conn(0x7f9c440f0440 msgr2=0x7f9c440fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 shutdown_connections 2021-11-20T07:42:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:06.215+0000 7f9c4ba80700 1 -- 172.21.15.71:0/2141321279 wait complete. 2021-11-20T07:42:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 71 2021-11-20T07:42:06.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:41:55.480859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 24373}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 63, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.112:6825/2992700860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2992700860}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2992700860}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:06.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:06.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: audit 2021-11-20T07:42:06.215180+0000 mon.smithi071 2021-11-20T07:42:06.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: (mon.0) 807 : audit [DBG] from='client.? 172.21.15.71:0/2141321279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:06.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: cluster 2021-11-20T07:42:06.280370+0000 mon.smithi071 (mon 2021-11-20T07:42:06.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: .0) 808 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:42:06.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: cluster 2021-11-20T07:42:06.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: 2021-11-20T07:42:06.283941+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:06.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:06 smithi112 conmon[36062]: 809 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T07:42:06.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:06 smithi071 conmon[33687]: audit 2021-11-20T07:42:06.215180+0000 mon.smithi071 (mon. 2021-11-20T07:42:06.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:06 smithi071 conmon[33687]: 0) 807 : audit [DBG] from='client.? 172.21.15.71:0/2141321279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:06.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:06 smithi071 conmon[33687]: cluster 2021-11-20T07:42:06.280370+0000 mon.smithi071 (mon.0) 808 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:42:06.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:06 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:42:06.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:06 smithi071 conmon[33687]: 11-20T07:42:06.283941+0000 mon.smithi071 (mon.0) 809 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T07:42:07.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/4197336033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f7c80 msgr2=0x7f6cf80f80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/4197336033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f7c80 0x7f6cf80f80a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f6ce8004660 tx=0x7f6ce8009b30).stop 2021-11-20T07:42:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/4197336033 shutdown_connections 2021-11-20T07:42:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/4197336033 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f6290 0x7f6cf80f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/4197336033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f7c80 0x7f6cf80f80a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.355+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/4197336033 >> 172.21.15.71:0/4197336033 conn(0x7f6cf80f1d10 msgr2=0x7f6cf80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/4197336033 shutdown_connections 2021-11-20T07:42:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/4197336033 wait complete. 2021-11-20T07:42:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 Processor -- start 2021-11-20T07:42:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 -- start start 2021-11-20T07:42:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:07.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 0x7f6cf81073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:07.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8107a00 con 0x7f6cf80f6290 2021-11-20T07:42:07.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cfcbb1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8108170 con 0x7f6cf80f7c80 2021-11-20T07:42:07.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:07.363 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.356+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49296/0 (socket says 172.21.15.71:49296) 2021-11-20T07:42:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf77fe700 1 -- 172.21.15.71:0/3287679451 learned_addr learned my addr 172.21.15.71:0/3287679451 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf6ffd700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 0x7f6cf81073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:07.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf77fe700 1 -- 172.21.15.71:0/3287679451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 msgr2=0x7f6cf81073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:07.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf77fe700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 0x7f6cf81073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:07.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf77fe700 1 -- 172.21.15.71:0/3287679451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce8005040 con 0x7f6cf80f6290 2021-11-20T07:42:07.365 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf6ffd700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 0x7f6cf81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:42:07.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cf77fe700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f6ce8009f50 tx=0x7f6ce8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:07.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ce8018070 con 0x7f6cf80f6290 2021-11-20T07:42:07.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf81083f0 con 0x7f6cf80f6290 2021-11-20T07:42:07.366 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ce8013410 con 0x7f6cf80f6290 2021-11-20T07:42:07.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ce801c620 con 0x7f6cf80f6290 2021-11-20T07:42:07.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.357+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf8108950 con 0x7f6cf80f6290 2021-11-20T07:42:07.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.358+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf8045ad0 con 0x7f6cf80f6290 2021-11-20T07:42:07.367 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.358+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6ce801d420 con 0x7f6cf80f6290 2021-11-20T07:42:07.368 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.359+0000 7f6cdffff700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 0x7f6ce005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:07.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.359+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6ce8078230 con 0x7f6cf80f6290 2021-11-20T07:42:07.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.359+0000 7f6cf6ffd700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 0x7f6ce005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:07.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.359+0000 7f6cf6ffd700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 0x7f6ce005e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6cec00a7e0 tx=0x7f6cec008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:07.369 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.363+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ce804aa40 con 0x7f6cf80f6290 2021-11-20T07:42:07.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.551+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cf8027a00 con 0x7f6cf80f6290 2021-11-20T07:42:07.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.551+0000 7f6cdffff700 1 -- 172.21.15.71:0/3287679451 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f6ce804e060 con 0x7f6cf80f6290 2021-11-20T07:42:07.553 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:07.554 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:07.428510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:resolve","state_seq":7,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:resolve","state_seq":6,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:07.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 msgr2=0x7f6ce005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:07.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 0x7f6ce005e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6cec00a7e0 tx=0x7f6cec008040).stop 2021-11-20T07:42:07.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 msgr2=0x7f6cf8106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:07.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f6ce8009f50 tx=0x7f6ce8009b30).stop 2021-11-20T07:42:07.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 shutdown_connections 2021-11-20T07:42:07.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6ce005c300 0x7f6ce005e7c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:07.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cf80f6290 0x7f6cf8106ed0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:07.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 --2- 172.21.15.71:0/3287679451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6cf80f7c80 0x7f6cf81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:07.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 >> 172.21.15.71:0/3287679451 conn(0x7f6cf80f1d10 msgr2=0x7f6cf80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:07.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 shutdown_connections 2021-11-20T07:42:07.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:07.553+0000 7f6cfcbb1700 1 -- 172.21.15.71:0/3287679451 wait complete. 2021-11-20T07:42:07.558 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 74 2021-11-20T07:42:07.565 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster 2021-11-20T07:42:06.416405+0000 mgr.smithi071.ynqmwn 2021-11-20T07:42:07.565 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: (mgr.14162) 304 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:42:07.566 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster 2021-11-20T07:42:06.426920+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:07.566 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 810 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:boot 2021-11-20T07:42:07.566 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster 2021-11-20T07:42:07.566 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 2021-11-20T07:42:06.426990+0000 2021-11-20T07:42:07.567 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: mon.smithi071 (mon.0) 2021-11-20T07:42:07.567 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 811 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 1 2021-11-20T07:42:07.567 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:42:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 11-20T07:42:06.427318+0000 2021-11-20T07:42:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: mon.smithi071 (mon.0) 812 : cluster [DBG] 2021-11-20T07:42:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: fsmap cephfs:1/2 {0=cephfs.smithi071.zaejme=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:42:07.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:42:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 07:42:06.427523+0000 mon.smithi071 ( 2021-11-20T07:42:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: mon.0) 813 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:42:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster 2021-11-20T07:42:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 2021-11-20T07:42: 2021-11-20T07:42:07.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: 06.430199+0000 mon.smithi071 (mon.0) 814 : 2021-11-20T07:42:07.570 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:07 smithi071 conmon[33687]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 2 up:standby 2021-11-20T07:42:07.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:07.428510+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:07.571 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:resolve 2021-11-20T07:42:07.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: cluster 2021-11-20T07:42:06.416405+0000 mgr.smithi071.ynqmwn (mgr.14162) 304 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T07:42:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: cluster 2021-11-20T07:42:06.426920+0000 mon.smithi071 (mon.0) 810 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:boot 2021-11-20T07:42:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: cluster 2021-11-20T07:42:06.426990+0000 2021-11-20T07:42:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: mon.smithi071 (mon.0) 811 : cluster [INF] Standby daemon mds.cephfs.smithi071.xtyomb assigned to filesystem cephfs as rank 1 2021-11-20T07:42:07.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: cluster 2021-11-20T 2021-11-20T07:42:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: 07:42:06.427318+0000 mon.smithi071 ( 2021-11-20T07:42:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: mon.0) 812 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi071.zaejme=up:resolve} 3 up:standby, 1 failed 2021-11-20T07:42:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: audit 2021-11-20T07:42:07.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: 2021-11-20T07:42 2021-11-20T07:42:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: :06.427523+0000 mon.smithi071 (mon.0) 813 : 2021-11-20T07:42:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:42:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:42:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: 20T07:42:06.430199+0000 mon.smithi071 2021-11-20T07:42:07.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:07 smithi112 conmon[36062]: (mon.0) 814 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:replay} 2 up:standby 2021-11-20T07:42:08.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:08.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 -- 172.21.15.71:0/3237194097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 msgr2=0x7fcf340f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:08.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/3237194097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf340f6c00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fcf24004660 tx=0x7fcf24009b30).stop 2021-11-20T07:42:08.693 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 -- 172.21.15.71:0/3237194097 shutdown_connections 2021-11-20T07:42:08.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/3237194097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf340f6c00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:08.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/3237194097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf340f5500 0x7fcf340f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:08.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 -- 172.21.15.71:0/3237194097 >> 172.21.15.71:0/3237194097 conn(0x7fcf340f0c40 msgr2=0x7fcf340f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:08.694 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 -- 172.21.15.71:0/3237194097 shutdown_connections 2021-11-20T07:42:08.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 -- 172.21.15.71:0/3237194097 wait complete. 2021-11-20T07:42:08.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.689+0000 7fcf3aa34700 1 Processor -- start 2021-11-20T07:42:08.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 -- start start 2021-11-20T07:42:08.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf340f5500 0x7fcf34109d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:08.695 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34060540 con 0x7fcf340f67a0 2021-11-20T07:42:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34060680 con 0x7fcf340f5500 2021-11-20T07:42:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49316/0 (socket says 172.21.15.71:49316) 2021-11-20T07:42:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 -- 172.21.15.71:0/820998871 learned_addr learned my addr 172.21.15.71:0/820998871 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 -- 172.21.15.71:0/820998871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf340f5500 msgr2=0x7fcf34109d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf340f5500 0x7fcf34109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 -- 172.21.15.71:0/820998871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24005040 con 0x7fcf340f67a0 2021-11-20T07:42:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf39231700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fcf24009f50 tx=0x7fcf24009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf24018070 con 0x7fcf340f67a0 2021-11-20T07:42:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf24013410 con 0x7fcf340f67a0 2021-11-20T07:42:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3410c7a0 con 0x7fcf340f67a0 2021-11-20T07:42:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf2401c3c0 con 0x7fcf340f67a0 2021-11-20T07:42:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.690+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3410cd00 con 0x7fcf340f67a0 2021-11-20T07:42:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.691+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf34045ad0 con 0x7fcf340f67a0 2021-11-20T07:42:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.691+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcf2401d420 con 0x7fcf340f67a0 2021-11-20T07:42:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.692+0000 7fcf2a7fc700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 0x7fcf2005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.692+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fcf24078050 con 0x7fcf340f67a0 2021-11-20T07:42:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.692+0000 7fcf39a32700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 0x7fcf2005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.692+0000 7fcf39a32700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 0x7fcf2005e7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcf340601c0 tx=0x7fcf300099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.696+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf2404a860 con 0x7fcf340f67a0 2021-11-20T07:42:08.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: cluster 2021-11-20T07:42:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: 2021-11-20T07:42:07.436507+0000 mon.smithi071 (mon.0) 815 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:resolve 2021-11-20T07:42:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: cluster 2021-11-20T07 2021-11-20T07:42:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: :42:07.436556+0000 mon.smithi071 (mon.0) 816 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:42:08.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: audit 2021 2021-11-20T07:42:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: -11-20T07:42: 2021-11-20T07:42:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: 07.552585+0000 mon.smithi071 (mon.0) 817 : 2021-11-20T07:42:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/3287679451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:42:08.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: -11-20T07:42:08. 2021-11-20T07:42:08.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: 417097+0000 mgr.smithi071.ynqmwn (mgr.14162) 305 : cluster 2021-11-20T07:42:08.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:08 smithi112 conmon[36062]: [DBG] pgmap v280: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:42:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: cluster 2021-11-20T07:42:07.436507+0000 mon.smithi071 (mon.0) 815 : cluster [DBG] 2021-11-20T07:42:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:resolve 2021-11-20T07:42:08.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: cluster 2021-11-20T07:42:07.436556+0000 mon.smithi071 (mon.0) 816 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:resolve,1=cephfs.smithi071.xtyomb=up:resolve} 2 up:standby 2021-11-20T07:42:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: audit 2021-11-20T07:42:07.552585+0000 mon.smithi071 (mon.0) 817 : audit [DBG] 2021-11-20T07:42:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/3287679451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:42:08.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:08 smithi071 conmon[33687]: 11-20T07:42:08.417097+0000 mgr.smithi071.ynqmwn (mgr.14162) 305 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T07:42:08.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.880+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf34027a00 con 0x7fcf340f67a0 2021-11-20T07:42:08.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.880+0000 7fcf2a7fc700 1 -- 172.21.15.71:0/820998871 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fcf2404de80 con 0x7fcf340f67a0 2021-11-20T07:42:08.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:08.884 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:08.435233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:08.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 msgr2=0x7fcf2005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:08.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 0x7fcf2005e7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcf340601c0 tx=0x7fcf300099e0).stop 2021-11-20T07:42:08.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 msgr2=0x7fcf3410c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:08.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fcf24009f50 tx=0x7fcf24009b30).stop 2021-11-20T07:42:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 shutdown_connections 2021-11-20T07:42:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf340f5500 0x7fcf34109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcf2005c300 0x7fcf2005e7c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 --2- 172.21.15.71:0/820998871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf340f67a0 0x7fcf3410c250 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:08.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 >> 172.21.15.71:0/820998871 conn(0x7fcf340f0c40 msgr2=0x7fcf340fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:08.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 shutdown_connections 2021-11-20T07:42:08.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:08.882+0000 7fcf3aa34700 1 -- 172.21.15.71:0/820998871 wait complete. 2021-11-20T07:42:08.892 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 75 2021-11-20T07:42:08.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:08.435233+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:reconnect', 'state_seq': 11, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:08.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:reconnect 2021-11-20T07:42:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: cluster 2021-11-20T07:42:08.442002+0000 mon.smithi071 (mon.0) 818 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:reconnect 2021-11-20T07:42:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: cluster 2021-11-20T07:42:08.442053+0000 mon.smithi071 (mon.0) 819 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:reconnect 2021-11-20T07:42:09.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: cluster 2021-11-20T07:42:08. 2021-11-20T07:42:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: 442118+0000 mon.smithi071 (mon.0) 820 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:42:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: cluster 2021-11-20T07: 2021-11-20T07:42:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: 42:08.466138+0000 mds.cephfs.smithi071.zaejme (mds.0) 2021-11-20T07:42:09.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:42:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:42:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: 20T07:42:08.466141+0000 mds.cephfs.smithi071.xtyomb (mds 2021-11-20T07:42:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: .1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:42:09.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: audit 2021 2021-11-20T07:42:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: -11-20T07:42:08.881839+0000 mon.smithi071 2021-11-20T07:42:09.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:09 smithi112 conmon[36062]: (mon.0) 821 : audit [DBG] from='client.? 172.21.15.71:0/820998871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:09.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: cluster 2021-11-20T07:42:08.442002+0000 mon.smithi071 (mon.0) 818 2021-11-20T07:42:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:reconnect 2021-11-20T07:42:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: cluster 2021-11-20T07:42:08.442053 2021-11-20T07:42:09.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 819 : cluster [DBG] 2021-11-20T07:42:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:reconnect 2021-11-20T07:42:09.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: cluster 2021-11-20T07:42:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: 2021-11-20T07:42 2021-11-20T07:42:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: :08. 2021-11-20T07:42:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: 442118+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: 820 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:reconnect,1=cephfs.smithi071.xtyomb=up:reconnect} 2 up:standby 2021-11-20T07:42:09.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:42:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: 20T07:42:08.466138+0000 mds.cephfs.smithi071.zaejme 2021-11-20T07:42:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: (mds.0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:42:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:42:09.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: 11-20T07:42:08.466141+0000 2021-11-20T07:42:09.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: mds.cephfs.smithi071.xtyomb (mds.1) 1 : cluster [DBG] 2021-11-20T07:42:09.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0259996 2021-11-20T07:42:09.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: audit 2021 2021-11-20T07:42:09.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: -11-20T07:42:08 2021-11-20T07:42:09.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: .881839+0000 mon.smithi071 (mon.0) 821 : 2021-11-20T07:42:09.824 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:09 smithi071 conmon[33687]: audit [DBG] from='client.? 172.21.15.71:0/820998871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:09.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 -- 172.21.15.71:0/1453110688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 msgr2=0x7f47ec0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 --2- 172.21.15.71:0/1453110688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec0f59c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f47dc0099e0 tx=0x7f47dc004d10).stop 2021-11-20T07:42:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 -- 172.21.15.71:0/1453110688 shutdown_connections 2021-11-20T07:42:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 --2- 172.21.15.71:0/1453110688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 0x7f47ec0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 --2- 172.21.15.71:0/1453110688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec0f59c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 -- 172.21.15.71:0/1453110688 >> 172.21.15.71:0/1453110688 conn(0x7f47ec0f0ce0 msgr2=0x7f47ec0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 -- 172.21.15.71:0/1453110688 shutdown_connections 2021-11-20T07:42:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 -- 172.21.15.71:0/1453110688 wait complete. 2021-11-20T07:42:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.024+0000 7f47f1d33700 1 Processor -- start 2021-11-20T07:42:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 -- start start 2021-11-20T07:42:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 0x7f47ec106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec106ad0 con 0x7f47ec0f55a0 2021-11-20T07:42:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec106c10 con 0x7f47ec0f6840 2021-11-20T07:42:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49324/0 (socket says 172.21.15.71:49324) 2021-11-20T07:42:10.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 -- 172.21.15.71:0/27513338 learned_addr learned my addr 172.21.15.71:0/27513338 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:10.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eaffd700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 0x7f47ec106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:10.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 -- 172.21.15.71:0/27513338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 msgr2=0x7f47ec106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:10.032 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 0x7f47ec106490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:10.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 -- 172.21.15.71:0/27513338 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc009640 con 0x7f47ec0f55a0 2021-11-20T07:42:10.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47eb7fe700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f47dc005010 tx=0x7f47dc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:10.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47dc00e030 con 0x7f47ec0f55a0 2021-11-20T07:42:10.033 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec10b650 con 0x7f47ec0f55a0 2021-11-20T07:42:10.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.025+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47dc007dd0 con 0x7f47ec0f55a0 2021-11-20T07:42:10.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.026+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47dc018070 con 0x7f47ec0f55a0 2021-11-20T07:42:10.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.026+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec10bb30 con 0x7f47ec0f55a0 2021-11-20T07:42:10.034 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.026+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec027a00 con 0x7f47ec0f55a0 2021-11-20T07:42:10.035 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.027+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f47dc01dab0 con 0x7f47ec0f55a0 2021-11-20T07:42:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.027+0000 7f47e3fff700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 0x7f47d805e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.027+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f47dc078190 con 0x7f47ec0f55a0 2021-11-20T07:42:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.027+0000 7f47eaffd700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 0x7f47d805e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.028+0000 7f47eaffd700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 0x7f47d805e750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f47d400b790 tx=0x7f47d4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:10.037 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.031+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47dc04a940 con 0x7f47ec0f55a0 2021-11-20T07:42:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.218+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47ec0fddb0 con 0x7f47ec0f55a0 2021-11-20T07:42:10.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.220+0000 7f47e3fff700 1 -- 172.21.15.71:0/27513338 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f47dc04df60 con 0x7f47ec0f55a0 2021-11-20T07:42:10.222 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:10.222 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:09.440260+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:10.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 msgr2=0x7f47d805e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:10.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 0x7f47d805e750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f47d400b790 tx=0x7f47d4008040).stop 2021-11-20T07:42:10.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 msgr2=0x7f47ec105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f47dc005010 tx=0x7f47dc004d10).stop 2021-11-20T07:42:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 shutdown_connections 2021-11-20T07:42:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47d805c290 0x7f47d805e750 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0x7f47d400b790 tx=0x7f47d4008040).stop 2021-11-20T07:42:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47ec0f55a0 0x7f47ec105fa0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 --2- 172.21.15.71:0/27513338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47ec0f6840 0x7f47ec106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.222+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 >> 172.21.15.71:0/27513338 conn(0x7f47ec0f0ce0 msgr2=0x7f47ec0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.223+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 shutdown_connections 2021-11-20T07:42:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:10.223+0000 7f47f1d33700 1 -- 172.21.15.71:0/27513338 wait complete. 2021-11-20T07:42:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 76 2021-11-20T07:42:10.239 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:09.440260+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:10.239 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:rejoin 2021-11-20T07:42:10.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: cluster 2021-11-20T07:42:09.447765+0000 mon.smithi071 (mon.0) 822 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:rejoin 2021-11-20T07:42:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: cluster 2021-11-20T07:42:09.447836+0000 mon.smithi071 (mon.0) 823 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:rejoin 2021-11-20T07:42:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: cluster 2021-11-20T07:42:09.447921+0000 mon.smithi071 (mon.0) 824 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:42:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: cluster 2021-11-20T07:42:09.448992+0000 2021-11-20T07:42:10.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: mon.smithi071 (mon.0) 825 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 1 2021-11-20T07:42:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:42:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: 11-20T07:42:10.220355+0000 mon.smithi071 2021-11-20T07:42:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: (mon.0) 826 : audit [DBG] from='client.? 172.21.15.71:0/27513338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:42:10.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: 11-20T07:42:10.417456+0000 mgr.smithi071.ynqmwn 2021-11-20T07:42:10.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:10 smithi112 conmon[36062]: (mgr.14162) 306 : cluster [DBG] pgmap v281: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T07:42:10.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: cluster 2021-11-20T07:42:09. 2021-11-20T07:42:10.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: 447765+0000 mon.smithi071 (mon.0) 822 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:rejoin 2021-11-20T07:42:10.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: cluster 2021-11-20T07:42:09.447836+0000 mon.smithi071 (mon.0) 823 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:rejoin 2021-11-20T07:42:10.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: cluster 2021-11-20T07:42:09.447921+0000 mon.smithi071 (mon.0) 824 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:rejoin,1=cephfs.smithi071.xtyomb=up:rejoin} 2 up:standby 2021-11-20T07:42:10.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: cluster 2021-11-20T07:42:09.448992+0000 mon.smithi071 (mon.0) 825 : cluster [INF] daemon mds.cephfs.smithi071.xtyomb is now active in filesystem cephfs as rank 1 2021-11-20T07:42:10.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: audit 2021-11-20T07:42:10.220355+0000 mon.smithi071 (mon.0) 826 : audit [DBG] from='client.? 172.21.15.71:0/27513338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:10.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:10 smithi071 conmon[33687]: cluster 2021-11-20T07:42:10.417456+0000 mgr.smithi071.ynqmwn (mgr.14162) 306 : cluster [DBG] pgmap v281: 65 pgs: 65 active+clean; 5.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T07:42:11.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:11.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 -- 172.21.15.71:0/435210942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 msgr2=0x7f85640f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:11.373 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 --2- 172.21.15.71:0/435210942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f85640f5920 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8560004660 tx=0x7f8560009b30).stop 2021-11-20T07:42:11.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 -- 172.21.15.71:0/435210942 shutdown_connections 2021-11-20T07:42:11.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 --2- 172.21.15.71:0/435210942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f85640f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 --2- 172.21.15.71:0/435210942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f85640f5920 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.374 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 -- 172.21.15.71:0/435210942 >> 172.21.15.71:0/435210942 conn(0x7f85640f0c40 msgr2=0x7f85640f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:11.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 -- 172.21.15.71:0/435210942 shutdown_connections 2021-11-20T07:42:11.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.369+0000 7f856a6cf700 1 -- 172.21.15.71:0/435210942 wait complete. 2021-11-20T07:42:11.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 Processor -- start 2021-11-20T07:42:11.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 -- start start 2021-11-20T07:42:11.375 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:11.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f856410c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:11.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8564060540 con 0x7f85640f5500 2021-11-20T07:42:11.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f856a6cf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8564060680 con 0x7f85640f67a0 2021-11-20T07:42:11.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:11.376 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49342/0 (socket says 172.21.15.71:49342) 2021-11-20T07:42:11.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 -- 172.21.15.71:0/1018421384 learned_addr learned my addr 172.21.15.71:0/1018421384 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:11.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f8568ecc700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f856410c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:11.377 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 -- 172.21.15.71:0/1018421384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 msgr2=0x7f856410c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:11.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f856410c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 -- 172.21.15.71:0/1018421384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560005040 con 0x7f85640f5500 2021-11-20T07:42:11.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f8568ecc700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f856410c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:11.378 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.370+0000 7f85696cd700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f8560004e30 tx=0x7f8560009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:11.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.371+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8560018070 con 0x7f85640f5500 2021-11-20T07:42:11.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.371+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856410c7e0 con 0x7f85640f5500 2021-11-20T07:42:11.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.371+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8560013410 con 0x7f85640f5500 2021-11-20T07:42:11.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.371+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856001c3c0 con 0x7f85640f5500 2021-11-20T07:42:11.379 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.371+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856410cd40 con 0x7f85640f5500 2021-11-20T07:42:11.380 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.372+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8564045ad0 con 0x7f85640f5500 2021-11-20T07:42:11.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.372+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f856001d420 con 0x7f85640f5500 2021-11-20T07:42:11.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.372+0000 7f8559ffb700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 0x7f855005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:11.381 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.372+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8560078010 con 0x7f85640f5500 2021-11-20T07:42:11.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.372+0000 7f8568ecc700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 0x7f855005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:11.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.373+0000 7f8568ecc700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 0x7f855005e7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8554009740 tx=0x7f8554009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:11.382 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.376+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f856004a820 con 0x7f85640f5500 2021-11-20T07:42:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:11 smithi071 conmon[33687]: cluster 2021-11-20T07:42:10.450550+0000 mon.smithi071 (mon.0) 827 2021-11-20T07:42:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:11 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:clientreplay 2021-11-20T07:42:11.490 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:11 smithi071 conmon[33687]: cluster 2021-11-20T07:42:10.450592+0000 mon.smithi071 (mon.0) 828 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:active 2021-11-20T07:42:11.491 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:11 smithi071 conmon[33687]: cluster 2021-11-20T07:42:10.450651+0000 mon.smithi071 (mon.0) 829 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:standby 2021-11-20T07:42:11.491 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:11 smithi071 conmon[33687]: cluster 2021-11-20T07:42:10.450707+0000 mon.smithi071 (mon.0) 830 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:clientreplay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:42:11.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.584+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8564027a00 con 0x7f85640f5500 2021-11-20T07:42:11.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.585+0000 7f8559ffb700 1 -- 172.21.15.71:0/1018421384 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f856004de40 con 0x7f85640f5500 2021-11-20T07:42:11.587 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:11.588 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:11.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.586+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 msgr2=0x7f855005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:11.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.586+0000 7f856a6cf700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 0x7f855005e7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8554009740 tx=0x7f8554009be0).stop 2021-11-20T07:42:11.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 msgr2=0x7f8564109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:11.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f8560004e30 tx=0x7f8560009b30).stop 2021-11-20T07:42:11.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 shutdown_connections 2021-11-20T07:42:11.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c300 0x7f855005e7c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f5500 0x7f8564109d90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 --2- 172.21.15.71:0/1018421384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f67a0 0x7f856410c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:11.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 >> 172.21.15.71:0/1018421384 conn(0x7f85640f0c40 msgr2=0x7f85640fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:11.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 shutdown_connections 2021-11-20T07:42:11.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:11.587+0000 7f856a6cf700 1 -- 172.21.15.71:0/1018421384 wait complete. 2021-11-20T07:42:11.591 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:11.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:11.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:11.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:11 smithi112 conmon[36062]: cluster 2021-11-20T07:42:10.450550+0000 mon.smithi071 (mon.0) 827 : cluster [DBG] mds.? [v2:172.21.15.71:6828/256674091,v1:172.21.15.71:6829/256674091] up:clientreplay 2021-11-20T07:42:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:11 smithi112 conmon[36062]: cluster 2021-11-20T07:42:10.450592+0000 mon.smithi071 (mon.0) 828 : cluster [DBG] mds.? [v2:172.21.15.71:6826/245260831,v1:172.21.15.71:6827/245260831] up:active 2021-11-20T07:42:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:11 smithi112 conmon[36062]: cluster 2021-11-20T07:42:10.450651+0000 mon.smithi071 (mon.0) 829 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:standby 2021-11-20T07:42:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:11 smithi112 conmon[36062]: cluster 2021-11-20T07:42:10.450707+0000 mon.smithi071 (mon.0 2021-11-20T07:42:11.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:11 smithi112 conmon[36062]: ) 830 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi071.zaejme=up:clientreplay,1=cephfs.smithi071.xtyomb=up:active} 2 up:standby 2021-11-20T07:42:12.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 -- 172.21.15.71:0/702332325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f6ca0 msgr2=0x7f88e80f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 --2- 172.21.15.71:0/702332325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f6ca0 0x7f88e80f70c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f88dc0046e0 tx=0x7f88dc009b30).stop 2021-11-20T07:42:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 -- 172.21.15.71:0/702332325 shutdown_connections 2021-11-20T07:42:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 --2- 172.21.15.71:0/702332325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f5950 0x7f88e80f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 --2- 172.21.15.71:0/702332325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f6ca0 0x7f88e80f70c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 -- 172.21.15.71:0/702332325 >> 172.21.15.71:0/702332325 conn(0x7f88e80f04a0 msgr2=0x7f88e80f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:12.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 -- 172.21.15.71:0/702332325 shutdown_connections 2021-11-20T07:42:12.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.728+0000 7f88eed84700 1 -- 172.21.15.71:0/702332325 wait complete. 2021-11-20T07:42:12.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 Processor -- start 2021-11-20T07:42:12.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 -- start start 2021-11-20T07:42:12.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:12.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 0x7f88e810dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:12.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e810e070 con 0x7f88e80f5950 2021-11-20T07:42:12.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.729+0000 7f88eed84700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e810e1b0 con 0x7f88e80f6ca0 2021-11-20T07:42:12.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:12.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49356/0 (socket says 172.21.15.71:49356) 2021-11-20T07:42:12.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 -- 172.21.15.71:0/3119855288 learned_addr learned my addr 172.21.15.71:0/3119855288 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:12.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88e7fff700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 0x7f88e810dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:12.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 -- 172.21.15.71:0/3119855288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 msgr2=0x7f88e810dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:12.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 0x7f88e810dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 -- 172.21.15.71:0/3119855288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc005040 con 0x7f88e80f5950 2021-11-20T07:42:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88e7fff700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 0x7f88e810dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88ecb20700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f88dc000c00 tx=0x7f88dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88dc007ba0 con 0x7f88e80f5950 2021-11-20T07:42:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88dc007d00 con 0x7f88e80f5950 2021-11-20T07:42:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e8112bf0 con 0x7f88e80f5950 2021-11-20T07:42:12.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88dc0223b0 con 0x7f88e80f5950 2021-11-20T07:42:12.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.730+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e8113130 con 0x7f88e80f5950 2021-11-20T07:42:12.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.731+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e8027a00 con 0x7f88e80f5950 2021-11-20T07:42:12.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.731+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f88dc02a070 con 0x7f88e80f5950 2021-11-20T07:42:12.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.732+0000 7f88e57fa700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 0x7f88d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:12.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.732+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f88dc077bb0 con 0x7f88e80f5950 2021-11-20T07:42:12.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.732+0000 7f88e7fff700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 0x7f88d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:12.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.732+0000 7f88e7fff700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 0x7f88d805e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f88d4009740 tx=0x7f88d4009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:12.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.735+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88dc07e080 con 0x7f88e80f5950 2021-11-20T07:42:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:12 smithi112 conmon[36062]: audit 2021-11-20T07:42:11.586185+0000 mon.smithi071 (mon.0) 831 : audit [DBG] from='client.? 172.21.15.71:0/1018421384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:12.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:12 smithi112 conmon[36062]: cluster 2021-11-20T07:42:12.417934+0000 mgr.smithi071.ynqmwn (mgr.14162) 307 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 5.5 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:42:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:12 smithi071 conmon[33687]: audit 2021-11-20T07:42:11.586185+0000 mon.smithi071 (mon.0) 831 : audit [DBG] from='client.? 172.21.15.71:0/1018421384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:12.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:12 smithi071 conmon[33687]: cluster 2021-11-20T07:42:12.417934+0000 mgr.smithi071.ynqmwn (mgr.14162) 307 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 5.5 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-20T07:42:12.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.920+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88e80fcd30 con 0x7f88e80f5950 2021-11-20T07:42:12.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.921+0000 7f88e57fa700 1 -- 172.21.15.71:0/3119855288 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f88dc01eba0 con 0x7f88e80f5950 2021-11-20T07:42:12.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:12.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:12.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 msgr2=0x7f88d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:12.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 0x7f88d805e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f88d4009740 tx=0x7f88d4009be0).stop 2021-11-20T07:42:12.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 msgr2=0x7f88e810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:12.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f88dc000c00 tx=0x7f88dc009b30).stop 2021-11-20T07:42:12.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 shutdown_connections 2021-11-20T07:42:12.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f88d805c2a0 0x7f88d805e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88e80f5950 0x7f88e810d5d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 --2- 172.21.15.71:0/3119855288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88e80f6ca0 0x7f88e810dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:12.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 >> 172.21.15.71:0/3119855288 conn(0x7f88e80f04a0 msgr2=0x7f88e80fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:12.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 shutdown_connections 2021-11-20T07:42:12.928 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:12.922+0000 7f88eed84700 1 -- 172.21.15.71:0/3119855288 wait complete. 2021-11-20T07:42:12.928 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:12.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:12.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:13.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:13 smithi112 conmon[36062]: audit 2021-11-20T07:42:12.921936+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:13.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:13 smithi112 conmon[36062]: 832 : audit [DBG] from='client.? 172.21.15.71:0/3119855288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:13 smithi071 conmon[33687]: audit 2021-11-20T07:42:12.921936+0000 mon.smithi071 (mon.0) 832 2021-11-20T07:42:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:13 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/3119855288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:13.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:14.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- 172.21.15.71:0/4179603996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f55b0 msgr2=0x7f19580f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:14.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 --2- 172.21.15.71:0/4179603996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f55b0 0x7f19580f59d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1954004660 tx=0x7f1954009b30).stop 2021-11-20T07:42:14.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- 172.21.15.71:0/4179603996 shutdown_connections 2021-11-20T07:42:14.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 --2- 172.21.15.71:0/4179603996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f6850 0x7f19580f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 --2- 172.21.15.71:0/4179603996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f55b0 0x7f19580f59d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- 172.21.15.71:0/4179603996 >> 172.21.15.71:0/4179603996 conn(0x7f19580f0d10 msgr2=0x7f19580f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:14.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- 172.21.15.71:0/4179603996 shutdown_connections 2021-11-20T07:42:14.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- 172.21.15.71:0/4179603996 wait complete. 2021-11-20T07:42:14.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 Processor -- start 2021-11-20T07:42:14.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- start start 2021-11-20T07:42:14.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f55b0 0x7f1958060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:14.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:14.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195810c1d0 con 0x7f19580f6850 2021-11-20T07:42:14.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.062+0000 7f196078c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195810c310 con 0x7f19580f55b0 2021-11-20T07:42:14.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:14.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49370/0 (socket says 172.21.15.71:49370) 2021-11-20T07:42:14.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 -- 172.21.15.71:0/2148312131 learned_addr learned my addr 172.21.15.71:0/2148312131 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:14.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195e528700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f55b0 0x7f1958060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:14.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 -- 172.21.15.71:0/2148312131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f55b0 msgr2=0x7f1958060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:14.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f55b0 0x7f1958060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 -- 172.21.15.71:0/2148312131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954005040 con 0x7f19580f6850 2021-11-20T07:42:14.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f195dd27700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1948002700 tx=0x7f1948009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:14.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1948009320 con 0x7f19580f6850 2021-11-20T07:42:14.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195810c570 con 0x7f19580f6850 2021-11-20T07:42:14.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f194800f040 con 0x7f19580f6850 2021-11-20T07:42:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19480147e0 con 0x7f19580f6850 2021-11-20T07:42:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.063+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195810cb30 con 0x7f19580f6850 2021-11-20T07:42:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.064+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1958027a00 con 0x7f19580f6850 2021-11-20T07:42:14.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.064+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1948009480 con 0x7f19580f6850 2021-11-20T07:42:14.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.065+0000 7f194effd700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 0x7f194405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:14.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.065+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f19480769b0 con 0x7f19580f6850 2021-11-20T07:42:14.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.065+0000 7f195e528700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 0x7f194405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:14.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.065+0000 7f195e528700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 0x7f194405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1954009f50 tx=0x7f1954009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:14.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.068+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19480491c0 con 0x7f19580f6850 2021-11-20T07:42:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.251+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19580fdd40 con 0x7f19580f6850 2021-11-20T07:42:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.252+0000 7f194effd700 1 -- 172.21.15.71:0/2148312131 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f194804c7e0 con 0x7f19580f6850 2021-11-20T07:42:14.254 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:14.254 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 msgr2=0x7f194405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 0x7f194405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1954009f50 tx=0x7f1954009b30).stop 2021-11-20T07:42:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 msgr2=0x7f195810bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f1948002700 tx=0x7f1948009de0).stop 2021-11-20T07:42:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 shutdown_connections 2021-11-20T07:42:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19580f55b0 0x7f1958060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f194405c2a0 0x7f194405e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 --2- 172.21.15.71:0/2148312131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19580f6850 0x7f195810bce0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 >> 172.21.15.71:0/2148312131 conn(0x7f19580f0d10 msgr2=0x7f19580fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 shutdown_connections 2021-11-20T07:42:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:14.253+0000 7f196078c700 1 -- 172.21.15.71:0/2148312131 wait complete. 2021-11-20T07:42:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:14.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:14.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:14.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:14 smithi071 conmon[33687]: audit 2021-11-20T07:42:14.252959+0000 mon.smithi071 (mon. 2021-11-20T07:42:14.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:14 smithi071 conmon[33687]: 0) 833 : audit [DBG] from='client.? 172.21.15.71:0/2148312131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:14.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:14 smithi071 conmon[33687]: cluster 2021-11-20T07:42:14.418492+0000 mgr.smithi071.ynqmwn (mgr.14162) 308 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:14 smithi112 conmon[36062]: audit 2021-11-20T07:42:14.252959+0000 mon.smithi071 (mon.0) 833 : 2021-11-20T07:42:14.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:14 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/2148312131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:14.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:14 smithi112 conmon[36062]: cluster 2021-11-20T07:42:14.418492+0000 mgr.smithi071.ynqmwn (mgr.14162) 308 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:15.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:15.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- 172.21.15.71:0/1768574571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f00f9760 msgr2=0x7f07f00fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:15.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 --2- 172.21.15.71:0/1768574571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f00f9760 0x7f07f00fbb90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f07e4004660 tx=0x7f07e4009b30).stop 2021-11-20T07:42:15.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- 172.21.15.71:0/1768574571 shutdown_connections 2021-11-20T07:42:15.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 --2- 172.21.15.71:0/1768574571 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f00f9760 0x7f07f00fbb90 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 --2- 172.21.15.71:0/1768574571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f0060370 0x7f07f00f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- 172.21.15.71:0/1768574571 >> 172.21.15.71:0/1768574571 conn(0x7f07f00f0c10 msgr2=0x7f07f00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:15.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- 172.21.15.71:0/1768574571 shutdown_connections 2021-11-20T07:42:15.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- 172.21.15.71:0/1768574571 wait complete. 2021-11-20T07:42:15.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 Processor -- start 2021-11-20T07:42:15.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.391+0000 7f07f564f700 1 -- start start 2021-11-20T07:42:15.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:15.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 0x7f07f0107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:15.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f0107880 con 0x7f07f0060370 2021-11-20T07:42:15.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f01080c0 con 0x7f07f00f9760 2021-11-20T07:42:15.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:15.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49380/0 (socket says 172.21.15.71:49380) 2021-11-20T07:42:15.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 -- 172.21.15.71:0/2474929608 learned_addr learned my addr 172.21.15.71:0/2474929608 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07ef7fe700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 0x7f07f0107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 -- 172.21.15.71:0/2474929608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 msgr2=0x7f07f0107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 0x7f07f0107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 -- 172.21.15.71:0/2474929608 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e4005040 con 0x7f07f0060370 2021-11-20T07:42:15.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07ef7fe700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 0x7f07f0107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:15.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07effff700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f07e000c530 tx=0x7f07e0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:15.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07e000f650 con 0x7f07f0060370 2021-11-20T07:42:15.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07e000fc60 con 0x7f07f0060370 2021-11-20T07:42:15.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f01083a0 con 0x7f07f0060370 2021-11-20T07:42:15.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07e0012c80 con 0x7f07f0060370 2021-11-20T07:42:15.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.392+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f01088b0 con 0x7f07f0060370 2021-11-20T07:42:15.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.393+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f00f48c0 con 0x7f07f0060370 2021-11-20T07:42:15.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.394+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f07e0010070 con 0x7f07f0060370 2021-11-20T07:42:15.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.394+0000 7f07ecff9700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 0x7f07d805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:15.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.394+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f07e00760f0 con 0x7f07f0060370 2021-11-20T07:42:15.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.394+0000 7f07ef7fe700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 0x7f07d805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:15.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.394+0000 7f07ef7fe700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 0x7f07d805e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f07e4009f50 tx=0x7f07e4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:15.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.398+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07e00452b0 con 0x7f07f0060370 2021-11-20T07:42:15.585 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.583+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f0045b50 con 0x7f07f0060370 2021-11-20T07:42:15.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.584+0000 7f07ecff9700 1 -- 172.21.15.71:0/2474929608 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f07f0045b50 con 0x7f07f0060370 2021-11-20T07:42:15.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:15.587 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:15.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 msgr2=0x7f07d805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:15.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 0x7f07d805e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f07e4009f50 tx=0x7f07e4009b30).stop 2021-11-20T07:42:15.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 msgr2=0x7f07f0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:15.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f07e000c530 tx=0x7f07e0007e20).stop 2021-11-20T07:42:15.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 shutdown_connections 2021-11-20T07:42:15.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f07d805c300 0x7f07d805e7c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07f0060370 0x7f07f0106d90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 --2- 172.21.15.71:0/2474929608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f07f00f9760 0x7f07f0107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:15.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 >> 172.21.15.71:0/2474929608 conn(0x7f07f00f0c10 msgr2=0x7f07f00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:15.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 shutdown_connections 2021-11-20T07:42:15.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:15.586+0000 7f07f564f700 1 -- 172.21.15.71:0/2474929608 wait complete. 2021-11-20T07:42:15.590 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:15.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:15.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:16.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:16.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.726+0000 7fb4a71e4700 1 -- 172.21.15.71:0/4111423478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 msgr2=0x7fb4a00f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:16.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.726+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/4111423478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a00f7940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fb490004660 tx=0x7fb490009b30).stop 2021-11-20T07:42:16.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- 172.21.15.71:0/4111423478 shutdown_connections 2021-11-20T07:42:16.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/4111423478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a00f7940 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:16.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/4111423478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a00f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:16.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- 172.21.15.71:0/4111423478 >> 172.21.15.71:0/4111423478 conn(0x7fb4a00f0320 msgr2=0x7fb4a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:16.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- 172.21.15.71:0/4111423478 shutdown_connections 2021-11-20T07:42:16.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- 172.21.15.71:0/4111423478 wait complete. 2021-11-20T07:42:16.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 Processor -- start 2021-11-20T07:42:16.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- start start 2021-11-20T07:42:16.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a0106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:16.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:16.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0107990 con 0x7fb4a00f51c0 2021-11-20T07:42:16.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a71e4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0108110 con 0x7fb4a00f4820 2021-11-20T07:42:16.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a59e1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:16.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a59e1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49394/0 (socket says 172.21.15.71:49394) 2021-11-20T07:42:16.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.727+0000 7fb4a59e1700 1 -- 172.21.15.71:0/1090058716 learned_addr learned my addr 172.21.15.71:0/1090058716 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:16.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a61e2700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a0106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:16.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a59e1700 1 -- 172.21.15.71:0/1090058716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 msgr2=0x7fb4a0106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a59e1700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a0106e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a59e1700 1 -- 172.21.15.71:0/1090058716 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb490005040 con 0x7fb4a00f51c0 2021-11-20T07:42:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a61e2700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a0106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a59e1700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb490009f50 tx=0x7fb490009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb490018070 con 0x7fb4a00f51c0 2021-11-20T07:42:16.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a0108330 con 0x7fb4a00f51c0 2021-11-20T07:42:16.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb490013410 con 0x7fb4a00f51c0 2021-11-20T07:42:16.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb49001c620 con 0x7fb4a00f51c0 2021-11-20T07:42:16.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.728+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a0108890 con 0x7fb4a00f51c0 2021-11-20T07:42:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.729+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0045ad0 con 0x7fb4a00f51c0 2021-11-20T07:42:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.729+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb49001d420 con 0x7fb4a00f51c0 2021-11-20T07:42:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.729+0000 7fb496ffd700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 0x7fb48c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.730+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb4900781e0 con 0x7fb4a00f51c0 2021-11-20T07:42:16.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.730+0000 7fb4a61e2700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 0x7fb48c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:16.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.730+0000 7fb4a61e2700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 0x7fb48c05e7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb4a0106b80 tx=0x7fb49c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:16.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.733+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb49004a9f0 con 0x7fb4a00f51c0 2021-11-20T07:42:16.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:16 smithi112 conmon[36062]: audit 2021-11-20T07:42:15.585407+0000 mon.smithi071 (mon.0) 834 : audit [DBG] from='client.? 172.21.15.71:0/2474929608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:16.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:16 smithi112 conmon[36062]: 2021-11-20T07:42:16.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:16 smithi071 conmon[33687]: audit 2021-11-20T07:42:15.585407+0000 mon.smithi071 (mon.0) 834 : audit [DBG] from='client.? 172.21.15.71:0/2474929608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:16.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.916+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4a0027a00 con 0x7fb4a00f51c0 2021-11-20T07:42:16.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.917+0000 7fb496ffd700 1 -- 172.21.15.71:0/1090058716 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7fb49004e010 con 0x7fb4a00f51c0 2021-11-20T07:42:16.920 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:16.921 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:16.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 msgr2=0x7fb48c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:16.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 0x7fb48c05e7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb4a0106b80 tx=0x7fb49c008040).stop 2021-11-20T07:42:16.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 msgr2=0x7fb4a0107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:16.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb490009f50 tx=0x7fb490009b30).stop 2021-11-20T07:42:16.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 shutdown_connections 2021-11-20T07:42:16.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb4a00f4820 0x7fb4a0106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:16.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb48c05c300 0x7fb48c05e7c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:16.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 --2- 172.21.15.71:0/1090058716 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a00f51c0 0x7fb4a0107300 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:16.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 >> 172.21.15.71:0/1090058716 conn(0x7fb4a00f0320 msgr2=0x7fb4a00fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:16.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 shutdown_connections 2021-11-20T07:42:16.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:16.919+0000 7fb4a71e4700 1 -- 172.21.15.71:0/1090058716 wait complete. 2021-11-20T07:42:16.927 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:16.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:16.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:17.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:17 smithi112 conmon[36062]: cluster 2021-11-20T07:42:16.418879+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:42:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:17 smithi112 conmon[36062]: mgr.14162) 309 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 404 B/s wr, 1 op/s 2021-11-20T07:42:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:17 smithi112 conmon[36062]: audit 2021-11-20T07:42:16.918435+0000 mon.smithi071 (mon.0) 835 : audit [DBG] from='client.? 172.21.15.71:0/1090058716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:17 smithi071 conmon[33687]: cluster 2021-11-20T07:42:16.418879+0000 mgr.smithi071.ynqmwn (mgr.14162) 309 : cluster [DBG] 2021-11-20T07:42:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:17 smithi071 conmon[33687]: pgmap v284: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 404 B/s wr, 1 op/s 2021-11-20T07:42:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:17 smithi071 conmon[33687]: audit 2021-11-20T07:42:16.918435+0000 mon.smithi071 (mon.0) 835 : audit [DBG] from='client.? 172.21.15.71:0/1090058716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:17.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:18.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 -- 172.21.15.71:0/3558107169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f5580 msgr2=0x7f31ec0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:18.062 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 --2- 172.21.15.71:0/3558107169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f5580 0x7f31ec0f59a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f31e8004660 tx=0x7f31e8009b30).stop 2021-11-20T07:42:18.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 -- 172.21.15.71:0/3558107169 shutdown_connections 2021-11-20T07:42:18.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 --2- 172.21.15.71:0/3558107169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f6820 0x7f31ec0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 --2- 172.21.15.71:0/3558107169 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f5580 0x7f31ec0f59a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 -- 172.21.15.71:0/3558107169 >> 172.21.15.71:0/3558107169 conn(0x7f31ec0f0ce0 msgr2=0x7f31ec0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:18.063 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 -- 172.21.15.71:0/3558107169 shutdown_connections 2021-11-20T07:42:18.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.057+0000 7f31f3588700 1 -- 172.21.15.71:0/3558107169 wait complete. 2021-11-20T07:42:18.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 Processor -- start 2021-11-20T07:42:18.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 -- start start 2021-11-20T07:42:18.064 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f5580 0x7f31ec060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:18.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:18.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec107ac0 con 0x7f31ec0f6820 2021-11-20T07:42:18.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f3588700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec10e5d0 con 0x7f31ec0f5580 2021-11-20T07:42:18.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:18.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49404/0 (socket says 172.21.15.71:49404) 2021-11-20T07:42:18.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 -- 172.21.15.71:0/3226857705 learned_addr learned my addr 172.21.15.71:0/3226857705 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:18.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f1324700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f5580 0x7f31ec060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:18.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 -- 172.21.15.71:0/3226857705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f5580 msgr2=0x7f31ec060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:18.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f5580 0x7f31ec060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 -- 172.21.15.71:0/3226857705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8005040 con 0x7f31ec0f6820 2021-11-20T07:42:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.058+0000 7f31f0b23700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f31dc0026d0 tx=0x7f31dc009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.059+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31dc009320 con 0x7f31ec0f6820 2021-11-20T07:42:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.059+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31dc00f040 con 0x7f31ec0f6820 2021-11-20T07:42:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.059+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec10e770 con 0x7f31ec0f6820 2021-11-20T07:42:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.059+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31dc0147e0 con 0x7f31ec0f6820 2021-11-20T07:42:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.059+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec10ed00 con 0x7f31ec0f6820 2021-11-20T07:42:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.060+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec046330 con 0x7f31ec0f6820 2021-11-20T07:42:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.060+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f31dc009480 con 0x7f31ec0f6820 2021-11-20T07:42:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.060+0000 7f31e1ffb700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 0x7f31d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:18.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.060+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f31dc076960 con 0x7f31ec0f6820 2021-11-20T07:42:18.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.060+0000 7f31f1324700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 0x7f31d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.061+0000 7f31f1324700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 0x7f31d805e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f31e8004e30 tx=0x7f31e8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.064+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31dc0491f0 con 0x7f31ec0f6820 2021-11-20T07:42:18.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.248+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31ec027a00 con 0x7f31ec0f6820 2021-11-20T07:42:18.250 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.248+0000 7f31e1ffb700 1 -- 172.21.15.71:0/3226857705 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f31dc04c810 con 0x7f31ec0f6820 2021-11-20T07:42:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:18.251 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:18.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 msgr2=0x7f31d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:18.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 0x7f31d805e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f31e8004e30 tx=0x7f31e8009b30).stop 2021-11-20T07:42:18.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 msgr2=0x7f31ec10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:18.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f31dc0026d0 tx=0x7f31dc009de0).stop 2021-11-20T07:42:18.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 shutdown_connections 2021-11-20T07:42:18.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec0f5580 0x7f31ec060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d805c2a0 0x7f31d805e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 --2- 172.21.15.71:0/3226857705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31ec0f6820 0x7f31ec10e0e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:18.254 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 >> 172.21.15.71:0/3226857705 conn(0x7f31ec0f0ce0 msgr2=0x7f31ec0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:18.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 shutdown_connections 2021-11-20T07:42:18.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:18.250+0000 7f31f3588700 1 -- 172.21.15.71:0/3226857705 wait complete. 2021-11-20T07:42:18.255 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:18.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:18.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:18.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:18 smithi071 conmon[33687]: audit 2021-11-20T07:42:18.249505+0000 mon.smithi071 (mon. 2021-11-20T07:42:18.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:18 smithi071 conmon[33687]: 0) 836 : audit [DBG] from='client.? 172.21.15.71:0/3226857705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:18.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:18 smithi112 conmon[36062]: audit 2021-11-20T07:42:18.249505+0000 mon.smithi071 (mon.0 2021-11-20T07:42:18.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:18 smithi112 conmon[36062]: ) 836 : audit [DBG] from='client.? 172.21.15.71:0/3226857705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:19.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:19.392 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2938157295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 msgr2=0x7f3cd00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:19.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2938157295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd00f6ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f3cc4004660 tx=0x7f3cc4009b30).stop 2021-11-20T07:42:19.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2938157295 shutdown_connections 2021-11-20T07:42:19.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2938157295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd00f6ca0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:19.393 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2938157295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3cd00f55a0 0x7f3cd00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:19.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2938157295 >> 172.21.15.71:0/2938157295 conn(0x7f3cd00f0ce0 msgr2=0x7f3cd00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:19.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2938157295 shutdown_connections 2021-11-20T07:42:19.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.388+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2938157295 wait complete. 2021-11-20T07:42:19.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 Processor -- start 2021-11-20T07:42:19.394 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 -- start start 2021-11-20T07:42:19.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3cd00f55a0 0x7f3cd0105dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:19.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:19.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0106900 con 0x7f3cd00f6840 2021-11-20T07:42:19.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cd59ef700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0106a40 con 0x7f3cd00f55a0 2021-11-20T07:42:19.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:19.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49416/0 (socket says 172.21.15.71:49416) 2021-11-20T07:42:19.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 -- 172.21.15.71:0/2751670284 learned_addr learned my addr 172.21.15.71:0/2751670284 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:19.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 -- 172.21.15.71:0/2751670284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3cd00f55a0 msgr2=0x7f3cd0105dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:19.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3cd00f55a0 0x7f3cd0105dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:19.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.389+0000 7f3cce7fc700 1 -- 172.21.15.71:0/2751670284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4005040 con 0x7f3cd00f6840 2021-11-20T07:42:19.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cce7fc700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3cc4009830 tx=0x7f3cc4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:19.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc401a070 con 0x7f3cd00f6840 2021-11-20T07:42:19.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cc4012c60 con 0x7f3cd00f6840 2021-11-20T07:42:19.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd005f450 con 0x7f3cd00f6840 2021-11-20T07:42:19.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cc401fa10 con 0x7f3cd00f6840 2021-11-20T07:42:19.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.390+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd005f9b0 con 0x7f3cd00f6840 2021-11-20T07:42:19.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.391+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd0046330 con 0x7f3cd00f6840 2021-11-20T07:42:19.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.391+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3cc40127b0 con 0x7f3cd00f6840 2021-11-20T07:42:19.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.391+0000 7f3cb77fe700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 0x7f3cb805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:19.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.391+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f3cc407a740 con 0x7f3cd00f6840 2021-11-20T07:42:19.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.391+0000 7f3cceffd700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 0x7f3cb805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:19.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.392+0000 7f3cceffd700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 0x7f3cb805e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3cd00f66a0 tx=0x7f3cc0006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:19.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.395+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cc404cf50 con 0x7f3cd00f6840 2021-11-20T07:42:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:19 smithi071 conmon[33687]: cluster 2021-11-20T07:42:18.419419+0000 mgr.smithi071.ynqmwn (mgr.14162) 310 : cluster [DBG] 2021-11-20T07:42:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:19 smithi071 conmon[33687]: pgmap v285: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:19.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:19 smithi071 conmon[33687]: audit 2021-11-20T07:42:18.939601+0000 mon.smithi071 (mon.0) 837 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:42:19.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:19 smithi071 conmon[33687]: audit 2021-11-20T07:42:18.940734+0000 mon.smithi071 (mon.0) 838 : audit [INF] 2021-11-20T07:42:19.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:19 smithi071 conmon[33687]: from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:42:19.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.584+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cd00ffdd0 con 0x7f3cd00f6840 2021-11-20T07:42:19.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.584+0000 7f3cb77fe700 1 -- 172.21.15.71:0/2751670284 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f3cc4050570 con 0x7f3cd00f6840 2021-11-20T07:42:19.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:19.588 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:19.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 msgr2=0x7f3cb805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:19.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 0x7f3cb805e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3cd00f66a0 tx=0x7f3cc0006ab0).stop 2021-11-20T07:42:19.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 msgr2=0x7f3cd01062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:19.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3cc4009830 tx=0x7f3cc4009b30).stop 2021-11-20T07:42:19.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 shutdown_connections 2021-11-20T07:42:19.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3cd00f55a0 0x7f3cd0105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:19.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3cb805c2a0 0x7f3cb805e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:19.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 --2- 172.21.15.71:0/2751670284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd00f6840 0x7f3cd01062c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:19.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 >> 172.21.15.71:0/2751670284 conn(0x7f3cd00f0ce0 msgr2=0x7f3cd00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:19.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 shutdown_connections 2021-11-20T07:42:19.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:19.586+0000 7f3cd59ef700 1 -- 172.21.15.71:0/2751670284 wait complete. 2021-11-20T07:42:19.592 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:19.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:19.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:19.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: cluster 2021-11-20T07:42:18.419419+0000 mgr.smithi071.ynqmwn (mgr.14162) 310 : cluster 2021-11-20T07:42:19.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: [DBG] pgmap v285: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:19.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: audit 2021-11-20T07:42:18.939601+0000 mon.smithi071 (mon.0) 837 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T07:42:19.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: audit 2021-11-20T07:42: 2021-11-20T07:42:19.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: 18.940734+0000 mon.smithi071 (mon.0) 838 : 2021-11-20T07:42:19.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:19 smithi112 conmon[36062]: audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi071.ynqmwn/trash_purge_schedule"}]: dispatch 2021-11-20T07:42:20.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:20.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 -- 172.21.15.71:0/424753582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f7960 msgr2=0x7fb9a00f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:20.728 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 --2- 172.21.15.71:0/424753582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f7960 0x7fb9a00f7dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb994004660 tx=0x7fb994009b30).stop 2021-11-20T07:42:20.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 -- 172.21.15.71:0/424753582 shutdown_connections 2021-11-20T07:42:20.729 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 --2- 172.21.15.71:0/424753582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f7960 0x7fb9a00f7dc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 --2- 172.21.15.71:0/424753582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f66c0 0x7fb9a00f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 -- 172.21.15.71:0/424753582 >> 172.21.15.71:0/424753582 conn(0x7fb9a00f1e20 msgr2=0x7fb9a00f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 -- 172.21.15.71:0/424753582 shutdown_connections 2021-11-20T07:42:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.724+0000 7fb9a5163700 1 -- 172.21.15.71:0/424753582 wait complete. 2021-11-20T07:42:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 Processor -- start 2021-11-20T07:42:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 -- start start 2021-11-20T07:42:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f7960 0x7fb9a0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a006bd70 con 0x7fb9a00f66c0 2021-11-20T07:42:20.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb9a5163700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a006a310 con 0x7fb9a00f7960 2021-11-20T07:42:20.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:20.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49422/0 (socket says 172.21.15.71:49422) 2021-11-20T07:42:20.732 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 -- 172.21.15.71:0/1716327129 learned_addr learned my addr 172.21.15.71:0/1716327129 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99e59c700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f7960 0x7fb9a0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 -- 172.21.15.71:0/1716327129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f7960 msgr2=0x7fb9a0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f7960 0x7fb9a0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.725+0000 7fb99ed9d700 1 -- 172.21.15.71:0/1716327129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb994005040 con 0x7fb9a00f66c0 2021-11-20T07:42:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb99ed9d700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb99000c530 tx=0x7fb990007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb99000f650 con 0x7fb9a00f66c0 2021-11-20T07:42:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a006a590 con 0x7fb9a00f66c0 2021-11-20T07:42:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99000fc60 con 0x7fb9a00f66c0 2021-11-20T07:42:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb990012d20 con 0x7fb9a00f66c0 2021-11-20T07:42:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.726+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a006ab50 con 0x7fb9a00f66c0 2021-11-20T07:42:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.727+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a0027a00 con 0x7fb9a00f66c0 2021-11-20T07:42:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.727+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb990019030 con 0x7fb9a00f66c0 2021-11-20T07:42:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.727+0000 7fb9877fe700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 0x7fb98805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.728+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb990075fe0 con 0x7fb9a00f66c0 2021-11-20T07:42:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.728+0000 7fb99e59c700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 0x7fb98805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.728+0000 7fb99e59c700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 0x7fb98805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fb994013580 tx=0x7fb994009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.732+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb990048840 con 0x7fb9a00f66c0 2021-11-20T07:42:20.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:20 smithi112 conmon[36062]: audit 2021-11-20T07:42:19.585480+0000 mon.smithi071 (mon.0) 839 : audit [DBG] from='client.? 172.21.15.71:0/2751670284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:20.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:20 smithi071 conmon[33687]: audit 2021-11-20T07:42:19.585480+0000 mon.smithi071 (mon.0) 839 2021-11-20T07:42:20.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:20 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/2751670284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:20.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.918+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a00fee50 con 0x7fb9a00f66c0 2021-11-20T07:42:20.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.919+0000 7fb9877fe700 1 -- 172.21.15.71:0/1716327129 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7fb990017090 con 0x7fb9a00f66c0 2021-11-20T07:42:20.921 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:20.921 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:20.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 msgr2=0x7fb98805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 0x7fb98805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fb994013580 tx=0x7fb994009b30).stop 2021-11-20T07:42:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 msgr2=0x7fb9a006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb99000c530 tx=0x7fb990007e10).stop 2021-11-20T07:42:20.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 shutdown_connections 2021-11-20T07:42:20.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fb98805c2a0 0x7fb98805e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9a00f66c0 0x7fb9a006b7c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 --2- 172.21.15.71:0/1716327129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9a00f7960 0x7fb9a0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:20.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 >> 172.21.15.71:0/1716327129 conn(0x7fb9a00f1e20 msgr2=0x7fb9a00fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:20.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 shutdown_connections 2021-11-20T07:42:20.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:20.920+0000 7fb9a5163700 1 -- 172.21.15.71:0/1716327129 wait complete. 2021-11-20T07:42:20.925 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:20.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:20.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:21.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:21 smithi112 conmon[36062]: cluster 2021-11-20T07:42:20.419836+0000 mgr.smithi071.ynqmwn (mgr.14162) 311 2021-11-20T07:42:21.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:21 smithi112 conmon[36062]: : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:21.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:21 smithi112 conmon[36062]: audit 2021-11-20T07:42:20.919950+0000 mon.smithi071 (mon.0) 840 : audit [DBG] from='client.? 172.21.15.71:0/1716327129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:21 smithi071 conmon[33687]: cluster 2021-11-20T07:42:20.419836+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:42:21.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:21 smithi071 conmon[33687]: ) 311 : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:21 smithi071 conmon[33687]: audit 2021-11-20T07:42:20.919950+0000 mon.smithi071 (mon.0) 840 : 2021-11-20T07:42:21.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:21 smithi071 conmon[33687]: audit [DBG] from='client.? 172.21.15.71:0/1716327129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:21.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:22.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 -- 172.21.15.71:0/376950555 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 msgr2=0x7ff21c0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:22.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 --2- 172.21.15.71:0/376950555 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c0f4c60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff210004660 tx=0x7ff210009b30).stop 2021-11-20T07:42:22.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 -- 172.21.15.71:0/376950555 shutdown_connections 2021-11-20T07:42:22.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 --2- 172.21.15.71:0/376950555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 0x7ff21c0f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 --2- 172.21.15.71:0/376950555 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c0f4c60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.060+0000 7ff221ba6700 1 -- 172.21.15.71:0/376950555 >> 172.21.15.71:0/376950555 conn(0x7ff21c0f0360 msgr2=0x7ff21c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:22.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 -- 172.21.15.71:0/376950555 shutdown_connections 2021-11-20T07:42:22.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 -- 172.21.15.71:0/376950555 wait complete. 2021-11-20T07:42:22.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 Processor -- start 2021-11-20T07:42:22.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 -- start start 2021-11-20T07:42:22.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:22.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 0x7ff21c1072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:22.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c1078f0 con 0x7ff21c0f4840 2021-11-20T07:42:22.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff221ba6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c108100 con 0x7ff21c0f51e0 2021-11-20T07:42:22.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.061+0000 7ff220ba4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:22.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49432/0 (socket says 172.21.15.71:49432) 2021-11-20T07:42:22.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 -- 172.21.15.71:0/2941177774 learned_addr learned my addr 172.21.15.71:0/2941177774 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:22.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff21bfff700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 0x7ff21c1072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:22.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 -- 172.21.15.71:0/2941177774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 msgr2=0x7ff21c1072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:22.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 0x7ff21c1072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 -- 172.21.15.71:0/2941177774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210005040 con 0x7ff21c0f4840 2021-11-20T07:42:22.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff220ba4700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ff210009f50 tx=0x7ff210009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:22.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff210018070 con 0x7ff21c0f4840 2021-11-20T07:42:22.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff210013410 con 0x7ff21c0f4840 2021-11-20T07:42:22.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21001c3c0 con 0x7ff21c0f4840 2021-11-20T07:42:22.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c108320 con 0x7ff21c0f4840 2021-11-20T07:42:22.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.062+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c108880 con 0x7ff21c0f4840 2021-11-20T07:42:22.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.063+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c0f3ff0 con 0x7ff21c0f4840 2021-11-20T07:42:22.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.063+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff21001d420 con 0x7ff21c0f4840 2021-11-20T07:42:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.064+0000 7ff2197fa700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 0x7ff20c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.064+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff210077ff0 con 0x7ff21c0f4840 2021-11-20T07:42:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.064+0000 7ff21bfff700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 0x7ff20c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.064+0000 7ff21bfff700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 0x7ff20c05e7c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff2080041d0 tx=0x7ff208009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:22.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.068+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff21004a880 con 0x7ff21c0f4840 2021-11-20T07:42:22.252 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.250+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff21c027a00 con 0x7ff21c0f4840 2021-11-20T07:42:22.253 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.251+0000 7ff2197fa700 1 -- 172.21.15.71:0/2941177774 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7ff21004dea0 con 0x7ff21c0f4840 2021-11-20T07:42:22.253 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:22.254 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:22.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.252+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 msgr2=0x7ff20c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:22.255 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.252+0000 7ff221ba6700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 0x7ff20c05e7c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff2080041d0 tx=0x7ff208009040).stop 2021-11-20T07:42:22.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.252+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 msgr2=0x7ff21c106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:22.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.252+0000 7ff221ba6700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ff210009f50 tx=0x7ff210009b30).stop 2021-11-20T07:42:22.256 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 shutdown_connections 2021-11-20T07:42:22.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff20c05c300 0x7ff20c05e7c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff21c0f4840 0x7ff21c106e00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 --2- 172.21.15.71:0/2941177774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff21c0f51e0 0x7ff21c1072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:22.257 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 >> 172.21.15.71:0/2941177774 conn(0x7ff21c0f0360 msgr2=0x7ff21c0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:22.258 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 shutdown_connections 2021-11-20T07:42:22.258 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:22.253+0000 7ff221ba6700 1 -- 172.21.15.71:0/2941177774 wait complete. 2021-11-20T07:42:22.258 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:22.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:22.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:22.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:22 smithi071 conmon[33687]: audit 2021-11-20T07:42:22.252352 2021-11-20T07:42:22.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:22 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.71:0/2941177774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:22.569 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:22 smithi071 conmon[33687]: cluster 2021-11-20T07:42:22.420264+0000 mgr.smithi071.ynqmwn (mgr.14162) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 341 B/s wr, 0 op/s 2021-11-20T07:42:22.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:22 smithi112 conmon[36062]: audit 2021-11-20T07:42:22.252352+0000 mon.smithi071 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.71:0/2941177774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:22.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:22 smithi112 conmon[36062]: cluster 2021-11-20T07:42:22.420264+0000 mgr.smithi071.ynqmwn (mgr.14162) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 341 B/s wr, 0 op/s 2021-11-20T07:42:23.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:23.395 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 -- 172.21.15.71:0/365989666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 msgr2=0x7f29dc0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:23.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 --2- 172.21.15.71:0/365989666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc0f7d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f29d8004660 tx=0x7f29d8009b30).stop 2021-11-20T07:42:23.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 -- 172.21.15.71:0/365989666 shutdown_connections 2021-11-20T07:42:23.396 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 --2- 172.21.15.71:0/365989666 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc0f7d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 --2- 172.21.15.71:0/365989666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 -- 172.21.15.71:0/365989666 >> 172.21.15.71:0/365989666 conn(0x7f29dc0f1de0 msgr2=0x7f29dc0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:23.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 -- 172.21.15.71:0/365989666 shutdown_connections 2021-11-20T07:42:23.397 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.392+0000 7f29e3969700 1 -- 172.21.15.71:0/365989666 wait complete. 2021-11-20T07:42:23.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 Processor -- start 2021-11-20T07:42:23.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 -- start start 2021-11-20T07:42:23.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:23.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:23.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc06bd70 con 0x7f29dc0f7920 2021-11-20T07:42:23.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e3969700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc06a310 con 0x7f29dc0f6680 2021-11-20T07:42:23.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:23.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49446/0 (socket says 172.21.15.71:49446) 2021-11-20T07:42:23.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 -- 172.21.15.71:0/3597278667 learned_addr learned my addr 172.21.15.71:0/3597278667 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:23.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e1705700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:23.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 -- 172.21.15.71:0/3597278667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 msgr2=0x7f29dc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:23.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 -- 172.21.15.71:0/3597278667 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8005040 con 0x7f29dc0f7920 2021-11-20T07:42:23.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e1705700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:23.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.393+0000 7f29e0f04700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f29d8000c00 tx=0x7f29d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:23.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.394+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d8019070 con 0x7f29dc0f7920 2021-11-20T07:42:23.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.394+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc06a530 con 0x7f29dc0f7920 2021-11-20T07:42:23.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.394+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29d8007e70 con 0x7f29dc0f7920 2021-11-20T07:42:23.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.394+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d8014d00 con 0x7f29dc0f7920 2021-11-20T07:42:23.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.394+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc06aa90 con 0x7f29dc0f7920 2021-11-20T07:42:23.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.395+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc046330 con 0x7f29dc0f7920 2021-11-20T07:42:23.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.395+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f29d8025070 con 0x7f29dc0f7920 2021-11-20T07:42:23.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.395+0000 7f29d1ffb700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 0x7f29c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:23.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.395+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f29d80773d0 con 0x7f29dc0f7920 2021-11-20T07:42:23.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.395+0000 7f29e1705700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 0x7f29c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:23.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.396+0000 7f29e1705700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 0x7f29c805e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f29dc0f7780 tx=0x7f29cc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:23.405 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.399+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29d8049be0 con 0x7f29dc0f7920 2021-11-20T07:42:23.589 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.587+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29dc027a00 con 0x7f29dc0f7920 2021-11-20T07:42:23.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.588+0000 7f29d1ffb700 1 -- 172.21.15.71:0/3597278667 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f29d804d200 con 0x7f29dc0f7920 2021-11-20T07:42:23.590 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:23.591 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:23.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.589+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 msgr2=0x7f29c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:23.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.589+0000 7f29e3969700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 0x7f29c805e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f29dc0f7780 tx=0x7f29cc00b040).stop 2021-11-20T07:42:23.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 msgr2=0x7f29dc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:23.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f29d8000c00 tx=0x7f29d800b040).stop 2021-11-20T07:42:23.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 shutdown_connections 2021-11-20T07:42:23.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29dc0f6680 0x7f29dc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.594 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f29c805c2a0 0x7f29c805e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.594 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 --2- 172.21.15.71:0/3597278667 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29dc0f7920 0x7f29dc069e20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:23.594 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 >> 172.21.15.71:0/3597278667 conn(0x7f29dc0f1de0 msgr2=0x7f29dc0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:23.594 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 shutdown_connections 2021-11-20T07:42:23.595 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:23.590+0000 7f29e3969700 1 -- 172.21.15.71:0/3597278667 wait complete. 2021-11-20T07:42:23.595 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:23.603 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:23.604 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:24.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:23 smithi112 conmon[36062]: audit 2021-11-20T07:42:23.589246+0000 mon.smithi071 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.71:0/3597278667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:24.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:23 smithi071 conmon[33687]: audit 2021-11-20T07:42:23.589246+0000 mon.smithi071 (mon. 2021-11-20T07:42:24.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:23 smithi071 conmon[33687]: 0) 842 : audit [DBG] from='client.? 172.21.15.71:0/3597278667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:24.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:24.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/1553976608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f6670 msgr2=0x7fa7e80f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:24.734 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/1553976608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f6670 0x7fa7e80f6a90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fa7d8004660 tx=0x7fa7d8009b30).stop 2021-11-20T07:42:24.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/1553976608 shutdown_connections 2021-11-20T07:42:24.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/1553976608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f7910 0x7fa7e80f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/1553976608 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f6670 0x7fa7e80f6a90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.735 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/1553976608 >> 172.21.15.71:0/1553976608 conn(0x7fa7e80f1db0 msgr2=0x7fa7e80f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:24.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/1553976608 shutdown_connections 2021-11-20T07:42:24.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/1553976608 wait complete. 2021-11-20T07:42:24.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 Processor -- start 2021-11-20T07:42:24.736 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.729+0000 7fa7ed3c6700 1 -- start start 2021-11-20T07:42:24.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f6670 0x7fa7e806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:24.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:24.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e806bd70 con 0x7fa7e80f7910 2021-11-20T07:42:24.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e806a310 con 0x7fa7e80f6670 2021-11-20T07:42:24.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:24.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49456/0 (socket says 172.21.15.71:49456) 2021-11-20T07:42:24.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 -- 172.21.15.71:0/4279742369 learned_addr learned my addr 172.21.15.71:0/4279742369 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:24.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e6ffd700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f6670 0x7fa7e806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:24.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 -- 172.21.15.71:0/4279742369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f6670 msgr2=0x7fa7e806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:24.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f6670 0x7fa7e806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 -- 172.21.15.71:0/4279742369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8005040 con 0x7fa7e80f7910 2021-11-20T07:42:24.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7e67fc700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa7dc00c790 tx=0x7fa7dc007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:24.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc002590 con 0x7fa7e80f7910 2021-11-20T07:42:24.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7dc002ba0 con 0x7fa7e80f7910 2021-11-20T07:42:24.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e806a590 con 0x7fa7e80f7910 2021-11-20T07:42:24.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc013d90 con 0x7fa7e80f7910 2021-11-20T07:42:24.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.730+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e806ab50 con 0x7fa7e80f7910 2021-11-20T07:42:24.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.731+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8046330 con 0x7fa7e80f7910 2021-11-20T07:42:24.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.732+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa7dc01a060 con 0x7fa7e80f7910 2021-11-20T07:42:24.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.732+0000 7fa7cf7fe700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 0x7fa7d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:24.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.732+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa7dc076610 con 0x7fa7e80f7910 2021-11-20T07:42:24.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.732+0000 7fa7e6ffd700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 0x7fa7d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:24.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.732+0000 7fa7e6ffd700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 0x7fa7d005e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa7d8004630 tx=0x7fa7d8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:24.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.736+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7dc048dc0 con 0x7fa7e80f7910 2021-11-20T07:42:24.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.920+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7e8027a00 con 0x7fa7e80f7910 2021-11-20T07:42:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.920+0000 7fa7cf7fe700 1 -- 172.21.15.71:0/4279742369 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2949 (secure 0 0 0) 0x7fa7dc04c3e0 con 0x7fa7e80f7910 2021-11-20T07:42:24.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:24.923 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":62},{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:10.445517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14754,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14754":{"gid":14754,"name":"cephfs.smithi071.zaejme","rank":0,"incarnation":69,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.71:6829/256674091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":256674091},{"type":"v1","addr":"172.21.15.71:6829","nonce":256674091}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 msgr2=0x7fa7d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 0x7fa7d005e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa7d8004630 tx=0x7fa7d8009b30).stop 2021-11-20T07:42:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 msgr2=0x7fa7e8069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa7dc00c790 tx=0x7fa7dc007d70).stop 2021-11-20T07:42:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 shutdown_connections 2021-11-20T07:42:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7e80f6670 0x7fa7e806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa7d005c2a0 0x7fa7d005e760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 --2- 172.21.15.71:0/4279742369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7e80f7910 0x7fa7e8069e20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 >> 172.21.15.71:0/4279742369 conn(0x7fa7e80f1db0 msgr2=0x7fa7e80fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 shutdown_connections 2021-11-20T07:42:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:24.922+0000 7fa7ed3c6700 1 -- 172.21.15.71:0/4279742369 wait complete. 2021-11-20T07:42:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2021-11-20T07:42:24.934 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:24 smithi071 conmon[33687]: cluster 2021-11-20T07:42:24.420598+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:42:24.935 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:24 smithi071 conmon[33687]: 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T07:42:24.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:10.445517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14754, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14754': {'gid': 14754, 'name': 'cephfs.smithi071.zaejme', 'rank': 0, 'incarnation': 69, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.71:6829/256674091', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 256674091}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 256674091}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:24.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14754:up:clientreplay 2021-11-20T07:42:25.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:24 smithi112 conmon[36062]: cluster 2021-11-20T07:42:24.420598+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:42:25.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:24 smithi112 conmon[36062]: .14162) 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T07:42:25.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:25.964 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: audit 2021-11-20T07:42:24.921769+0000 mon.smithi071 (mon.0) 843 : audit [DBG] from='client.? 172.21.15.71:0/4279742369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:25.965 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: cluster 2021-11-20T07:42:25.413083+0000 mon.smithi071 (mon.0) 844 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.zaejme as rank 0 with standby daemon mds.cephfs.smithi112.iqrmps 2021-11-20T07:42:25.965 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: cluster 2021-11-20T 2021-11-20T07:42:25.965 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: 07:42:25.413226+0000 mon.smithi071 (mon.0) 845 : cluster [INF] MDS daemon mds.cephfs.smithi071.zaejme is removed because it is dead or otherwise unavailable. 2021-11-20T07:42:25.965 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:42:25.966 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: 20T07:42:25.418272+0000 mon.smithi071 (mon 2021-11-20T07:42:25.966 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: .0) 846 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T07:42:25.966 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:42:25.966 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: 20T07:42:25.423184+0000 mon.smithi071 ( 2021-11-20T07:42:25.966 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:25 smithi071 conmon[33687]: mon.0) 847 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:42:26.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: audit 2021-11-20T07:42:24.921769+0000 mon.smithi071 (mon.0) 843 : audit 2021-11-20T07:42:26.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/4279742369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:26.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: cluster 2021-11-20T07:42:25.413083+0000 mon.smithi071 (mon.0) 844 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.zaejme as rank 0 with standby daemon mds.cephfs.smithi112.iqrmps 2021-11-20T07:42:26.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: cluster 2021-11-20T07:42:25.413226+0000 mon.smithi071 2021-11-20T07:42:26.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: (mon.0) 845 : cluster [INF] MDS daemon mds.cephfs.smithi071.zaejme is removed because it is dead or otherwise unavailable. 2021-11-20T07:42:26.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: cluster 2021-11-20T07:42:26.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: 2021-11-20T07:42:25.418272+0000 2021-11-20T07:42:26.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: mon.smithi071 (mon.0) 846 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T07:42:26.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: 2021-11-20T07:42:26.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: cluster 2021-11 2021-11-20T07:42:26.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: -20T07:42:25.423184+0000 2021-11-20T07:42:26.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: mon.smithi071 (mon.0) 847 : cluster [DBG] 2021-11-20T07:42:26.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:25 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:42:26.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- 172.21.15.71:0/2757201829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f59d0 msgr2=0x7fc4700f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:26.065 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 --2- 172.21.15.71:0/2757201829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5e50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc460004660 tx=0x7fc460009b30).stop 2021-11-20T07:42:26.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- 172.21.15.71:0/2757201829 shutdown_connections 2021-11-20T07:42:26.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 --2- 172.21.15.71:0/2757201829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5e50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 --2- 172.21.15.71:0/2757201829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f6d10 0x7fc4700f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.066 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- 172.21.15.71:0/2757201829 >> 172.21.15.71:0/2757201829 conn(0x7fc4700f04a0 msgr2=0x7fc4700f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:26.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- 172.21.15.71:0/2757201829 shutdown_connections 2021-11-20T07:42:26.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- 172.21.15.71:0/2757201829 wait complete. 2021-11-20T07:42:26.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 Processor -- start 2021-11-20T07:42:26.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.060+0000 7fc478094700 1 -- start start 2021-11-20T07:42:26.067 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47010c250 con 0x7fc4700f6d10 2021-11-20T07:42:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47010c390 con 0x7fc4700f59d0 2021-11-20T07:42:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49466/0 (socket says 172.21.15.71:49466) 2021-11-20T07:42:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 -- 172.21.15.71:0/3639425464 learned_addr learned my addr 172.21.15.71:0/3639425464 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc475e30700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 -- 172.21.15.71:0/3639425464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 msgr2=0x7fc4700f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 -- 172.21.15.71:0/3639425464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460005040 con 0x7fc4700f6d10 2021-11-20T07:42:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc475e30700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc47562f700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc460009830 tx=0x7fc460009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc460018070 con 0x7fc4700f6d10 2021-11-20T07:42:26.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc460013410 con 0x7fc4700f6d10 2021-11-20T07:42:26.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc46001c620 con 0x7fc4700f6d10 2021-11-20T07:42:26.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47010c590 con 0x7fc4700f6d10 2021-11-20T07:42:26.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.061+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47010cac0 con 0x7fc4700f6d10 2021-11-20T07:42:26.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.062+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc470046330 con 0x7fc4700f6d10 2021-11-20T07:42:26.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.063+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc46001c8e0 con 0x7fc4700f6d10 2021-11-20T07:42:26.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.063+0000 7fc4667fc700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 0x7fc45c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:26.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.063+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc4600782a0 con 0x7fc4700f6d10 2021-11-20T07:42:26.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.063+0000 7fc475e30700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 0x7fc45c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:26.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.063+0000 7fc475e30700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 0x7fc45c05e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc46c0041d0 tx=0x7fc46c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:26.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.067+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc46004aa00 con 0x7fc4700f6d10 2021-11-20T07:42:26.260 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.259+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc470027a00 con 0x7fc4700f6d10 2021-11-20T07:42:26.261 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.259+0000 7fc4667fc700 1 -- 172.21.15.71:0/3639425464 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7fc46004e020 con 0x7fc4700f6d10 2021-11-20T07:42:26.262 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:26.262 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:25.413728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:replay","state_seq":3,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:26.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 msgr2=0x7fc45c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:26.264 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 0x7fc45c05e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc46c0041d0 tx=0x7fc46c009040).stop 2021-11-20T07:42:26.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 msgr2=0x7fc47010bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:26.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc460009830 tx=0x7fc460009b30).stop 2021-11-20T07:42:26.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 shutdown_connections 2021-11-20T07:42:26.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4700f59d0 0x7fc4700f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc45c05c2a0 0x7fc45c05e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 --2- 172.21.15.71:0/3639425464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4700f6d10 0x7fc47010bd60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:26.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 >> 172.21.15.71:0/3639425464 conn(0x7fc4700f04a0 msgr2=0x7fc4700fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:26.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 shutdown_connections 2021-11-20T07:42:26.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:26.261+0000 7fc478094700 1 -- 172.21.15.71:0/3639425464 wait complete. 2021-11-20T07:42:26.268 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 78 2021-11-20T07:42:26.278 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:25.413728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:26.278 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:replay 2021-11-20T07:42:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: audit 2021-11-20T07:42:26.260713+0000 mon.smithi071 (mon. 2021-11-20T07:42:26.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: 0) 848 : audit [DBG] from='client.? 172.21.15.71:0/3639425464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: cluster 2021-11-20T07:42:26.421084+0000 mgr.smithi071.ynqmwn (mgr.14162) 314 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:42:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: cluster 2021-11-20T07:42:26.424489+0000 mon.smithi071 (mon.0) 849 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:resolve 2021-11-20T07:42:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: cluster 2021-11-20T07:42:26.424553+0000 mon.smithi071 (mon.0 2021-11-20T07:42:26.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:26 smithi071 conmon[33687]: ) 850 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:42:27.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: audit 2021-11-20T07:42:26.260713+0000 mon.smithi071 (mon.0) 848 : audit [DBG] from='client.? 172.21.15.71:0/3639425464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:27.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: cluster 2021-11-20T07:42:26.421084+0000 mgr.smithi071.ynqmwn (mgr.14162) 314 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:42:27.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: cluster 2021-11-20T07:42:26.424489+0000 mon.smithi071 (mon. 2021-11-20T07:42:27.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: 0) 849 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:resolve 2021-11-20T07:42:27.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: cluster 2021-11- 2021-11-20T07:42:27.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: 20T07:42:26.424553+0000 2021-11-20T07:42:27.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: mon.smithi071 (mon.0) 850 : cluster [DBG] 2021-11-20T07:42:27.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:26 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi071.xtyomb=up:active} 1 up:standby 2021-11-20T07:42:27.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:27.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 -- 172.21.15.71:0/53081214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f51e0 msgr2=0x7fab440f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:27.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 --2- 172.21.15.71:0/53081214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f51e0 0x7fab440f7960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fab30004660 tx=0x7fab30009b30).stop 2021-11-20T07:42:27.398 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 -- 172.21.15.71:0/53081214 shutdown_connections 2021-11-20T07:42:27.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 --2- 172.21.15.71:0/53081214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f51e0 0x7fab440f7960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 --2- 172.21.15.71:0/53081214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f4840 0x7fab440f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 -- 172.21.15.71:0/53081214 >> 172.21.15.71:0/53081214 conn(0x7fab440f0360 msgr2=0x7fab440f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:27.399 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 -- 172.21.15.71:0/53081214 shutdown_connections 2021-11-20T07:42:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.394+0000 7fab49cba700 1 -- 172.21.15.71:0/53081214 wait complete. 2021-11-20T07:42:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 Processor -- start 2021-11-20T07:42:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 -- start start 2021-11-20T07:42:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f51e0 0x7fab441072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441078b0 con 0x7fab440f4840 2021-11-20T07:42:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab49cba700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441080f0 con 0x7fab440f51e0 2021-11-20T07:42:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab48cb8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab48cb8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49478/0 (socket says 172.21.15.71:49478) 2021-11-20T07:42:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab48cb8700 1 -- 172.21.15.71:0/586996568 learned_addr learned my addr 172.21.15.71:0/586996568 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.395+0000 7fab43fff700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f51e0 0x7fab441072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab48cb8700 1 -- 172.21.15.71:0/586996568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f51e0 msgr2=0x7fab441072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab48cb8700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f51e0 0x7fab441072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab48cb8700 1 -- 172.21.15.71:0/586996568 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30005040 con 0x7fab440f4840 2021-11-20T07:42:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab48cb8700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fab3800c530 tx=0x7fab38007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab3800f650 con 0x7fab440f4840 2021-11-20T07:42:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab3800fc60 con 0x7fab440f4840 2021-11-20T07:42:27.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab441083d0 con 0x7fab440f4840 2021-11-20T07:42:27.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab38012d20 con 0x7fab440f4840 2021-11-20T07:42:27.404 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.396+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab44108910 con 0x7fab440f4840 2021-11-20T07:42:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.397+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab440f3ff0 con 0x7fab440f4840 2021-11-20T07:42:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.397+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fab38019030 con 0x7fab440f4840 2021-11-20T07:42:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.397+0000 7fab417fa700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 0x7fab3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.397+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fab38076070 con 0x7fab440f4840 2021-11-20T07:42:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.397+0000 7fab43fff700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 0x7fab3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.398+0000 7fab43fff700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 0x7fab3405e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fab44106b50 tx=0x7fab30009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.401+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab38048880 con 0x7fab440f4840 2021-11-20T07:42:27.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.586+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab440fee70 con 0x7fab440f4840 2021-11-20T07:42:27.588 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.586+0000 7fab417fa700 1 -- 172.21.15.71:0/586996568 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7fab38017070 con 0x7fab440f4840 2021-11-20T07:42:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:27.589 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:27.590 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 msgr2=0x7fab3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:27.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 0x7fab3405e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fab44106b50 tx=0x7fab30009b30).stop 2021-11-20T07:42:27.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 msgr2=0x7fab44106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:27.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fab3800c530 tx=0x7fab38007e10).stop 2021-11-20T07:42:27.591 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 shutdown_connections 2021-11-20T07:42:27.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fab3405c300 0x7fab3405e7c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab440f4840 0x7fab44106dc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 --2- 172.21.15.71:0/586996568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab440f51e0 0x7fab441072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:27.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 >> 172.21.15.71:0/586996568 conn(0x7fab440f0360 msgr2=0x7fab440fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:27.592 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 shutdown_connections 2021-11-20T07:42:27.593 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:27.588+0000 7fab49cba700 1 -- 172.21.15.71:0/586996568 wait complete. 2021-11-20T07:42:27.593 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:27.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:27.602 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:27.932 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:27 smithi071 conmon[33687]: audit 2021-11-20T07:42:27.587871+0000 mon.smithi071 (mon.0) 851 : audit [DBG] from='client.? 172.21.15.71:0/586996568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:28.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:27 smithi112 conmon[36062]: audit 2021-11-20T07:42:27.587871+0000 mon.smithi071 (mon.0) 851 : audit [DBG] from='client.? 172.21.15.71:0/586996568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:28.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:28.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3962038634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 msgr2=0x7ff8380f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:28.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3962038634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff8380f7d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ff82c004660 tx=0x7ff82c009b30).stop 2021-11-20T07:42:28.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3962038634 shutdown_connections 2021-11-20T07:42:28.737 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3962038634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff8380f7d80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3962038634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 0x7ff8380f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3962038634 >> 172.21.15.71:0/3962038634 conn(0x7ff8380f1de0 msgr2=0x7ff8380f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:28.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3962038634 shutdown_connections 2021-11-20T07:42:28.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3962038634 wait complete. 2021-11-20T07:42:28.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 Processor -- start 2021-11-20T07:42:28.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.731+0000 7ff83e7ef700 1 -- start start 2021-11-20T07:42:28.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 0x7ff83806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:28.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:28.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83806bd70 con 0x7ff8380f7920 2021-11-20T07:42:28.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83806a310 con 0x7ff8380f6680 2021-11-20T07:42:28.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:28.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49490/0 (socket says 172.21.15.71:49490) 2021-11-20T07:42:28.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 -- 172.21.15.71:0/3275933621 learned_addr learned my addr 172.21.15.71:0/3275933621 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:28.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff837fff700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 0x7ff83806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:28.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 -- 172.21.15.71:0/3275933621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 msgr2=0x7ff83806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:28.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 0x7ff83806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 -- 172.21.15.71:0/3275933621 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c005040 con 0x7ff8380f7920 2021-11-20T07:42:28.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff8377fe700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff82c009f50 tx=0x7ff82c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:28.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff82c018070 con 0x7ff8380f7920 2021-11-20T07:42:28.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83806a530 con 0x7ff8380f7920 2021-11-20T07:42:28.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff82c013410 con 0x7ff8380f7920 2021-11-20T07:42:28.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff82c01c3c0 con 0x7ff8380f7920 2021-11-20T07:42:28.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.732+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83806aa90 con 0x7ff8380f7920 2021-11-20T07:42:28.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.733+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff838046330 con 0x7ff8380f7920 2021-11-20T07:42:28.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.734+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff82c01d420 con 0x7ff8380f7920 2021-11-20T07:42:28.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.734+0000 7ff834ff9700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 0x7ff82005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:28.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.734+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7ff82c078040 con 0x7ff8380f7920 2021-11-20T07:42:28.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.734+0000 7ff837fff700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 0x7ff82005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:28.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.734+0000 7ff837fff700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 0x7ff82005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7ff8380f7780 tx=0x7ff82800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:28.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.738+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff82c04a800 con 0x7ff8380f7920 2021-11-20T07:42:28.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.922+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff838027a00 con 0x7ff8380f7920 2021-11-20T07:42:28.924 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.922+0000 7ff834ff9700 1 -- 172.21.15.71:0/3275933621 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7ff82c04de20 con 0x7ff8380f7920 2021-11-20T07:42:28.925 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:28.925 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:28.926 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 msgr2=0x7ff82005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:28.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 0x7ff82005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7ff8380f7780 tx=0x7ff82800b040).stop 2021-11-20T07:42:28.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 msgr2=0x7ff838069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:28.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff82c009f50 tx=0x7ff82c009b30).stop 2021-11-20T07:42:28.927 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 shutdown_connections 2021-11-20T07:42:28.928 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8380f6680 0x7ff83806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.928 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff82005c2a0 0x7ff82005e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.928 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 --2- 172.21.15.71:0/3275933621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8380f7920 0x7ff838069e20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:28.928 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 >> 172.21.15.71:0/3275933621 conn(0x7ff8380f1de0 msgr2=0x7ff8380fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:28.929 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 shutdown_connections 2021-11-20T07:42:28.929 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:28.924+0000 7ff83e7ef700 1 -- 172.21.15.71:0/3275933621 wait complete. 2021-11-20T07:42:28.929 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:28.936 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:28 smithi071 conmon[33687]: cluster 2021-11-20T07:42:28.421517+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:42:28.936 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:28 smithi071 conmon[33687]: ) 315 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:42:28.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:28.938 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:29.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:28 smithi112 conmon[36062]: cluster 2021-11-20T07:42:28.421517+0000 mgr.smithi071.ynqmwn (mgr.14162) 315 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:42:29.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:29.967 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:29 smithi071 conmon[33687]: audit 2021-11-20T07:42:28.923835+0000 mon.smithi071 (mon.0) 852 : audit [DBG] 2021-11-20T07:42:29.968 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:29 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/3275933621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:30.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:29 smithi112 conmon[36062]: audit 2021-11-20T07:42:28.923835+0000 mon.smithi071 (mon.0) 852 : audit [DBG] from='client.? 172.21.15.71:0/3275933621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:30.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 -- 172.21.15.71:0/2585359312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 msgr2=0x7f5cd00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 --2- 172.21.15.71:0/2585359312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd00f59a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f5ccc004660 tx=0x7f5ccc009b30).stop 2021-11-20T07:42:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 -- 172.21.15.71:0/2585359312 shutdown_connections 2021-11-20T07:42:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 --2- 172.21.15.71:0/2585359312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 0x7f5cd00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 --2- 172.21.15.71:0/2585359312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd00f59a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 -- 172.21.15.71:0/2585359312 >> 172.21.15.71:0/2585359312 conn(0x7f5cd00f0ce0 msgr2=0x7f5cd00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 -- 172.21.15.71:0/2585359312 shutdown_connections 2021-11-20T07:42:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.064+0000 7f5cd837b700 1 -- 172.21.15.71:0/2585359312 wait complete. 2021-11-20T07:42:30.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 Processor -- start 2021-11-20T07:42:30.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 -- start start 2021-11-20T07:42:30.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:30.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 0x7f5cd010bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:30.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd010c190 con 0x7f5cd00f5580 2021-11-20T07:42:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd837b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd010c2d0 con 0x7f5cd00f6820 2021-11-20T07:42:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49498/0 (socket says 172.21.15.71:49498) 2021-11-20T07:42:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 -- 172.21.15.71:0/1335801178 learned_addr learned my addr 172.21.15.71:0/1335801178 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd5916700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 0x7f5cd010bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 -- 172.21.15.71:0/1335801178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 msgr2=0x7f5cd010bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 0x7f5cd010bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.065+0000 7f5cd6117700 1 -- 172.21.15.71:0/1335801178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc005040 con 0x7f5cd00f5580 2021-11-20T07:42:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cd6117700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f5ccc009830 tx=0x7f5ccc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ccc018070 con 0x7f5cd00f5580 2021-11-20T07:42:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ccc013410 con 0x7f5cd00f5580 2021-11-20T07:42:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd010c4d0 con 0x7f5cd00f5580 2021-11-20T07:42:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ccc01c620 con 0x7f5cd00f5580 2021-11-20T07:42:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.066+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd010ca30 con 0x7f5cd00f5580 2021-11-20T07:42:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.067+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd0027a00 con 0x7f5cd00f5580 2021-11-20T07:42:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.067+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5ccc01d420 con 0x7f5cd00f5580 2021-11-20T07:42:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.067+0000 7f5cc6ffd700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 0x7f5cbc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.067+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5ccc078300 con 0x7f5cd00f5580 2021-11-20T07:42:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.067+0000 7f5cd5916700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 0x7f5cbc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.068+0000 7f5cd5916700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 0x7f5cbc05e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f5cc0007fd0 tx=0x7f5cc0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.071+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ccc04aac0 con 0x7f5cd00f5580 2021-11-20T07:42:30.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.261+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cd00fdd10 con 0x7f5cd00f5580 2021-11-20T07:42:30.263 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.261+0000 7f5cc6ffd700 1 -- 172.21.15.71:0/1335801178 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f5ccc04e0e0 con 0x7f5cd00f5580 2021-11-20T07:42:30.264 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:30.264 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:30.265 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 msgr2=0x7f5cbc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:30.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 0x7f5cbc05e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f5cc0007fd0 tx=0x7f5cc0009040).stop 2021-11-20T07:42:30.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 msgr2=0x7f5cd0060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:30.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f5ccc009830 tx=0x7f5ccc009b30).stop 2021-11-20T07:42:30.266 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 shutdown_connections 2021-11-20T07:42:30.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5cbc05c2a0 0x7f5cbc05e760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cd00f5580 0x7f5cd0060510 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.267 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 --2- 172.21.15.71:0/1335801178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cd00f6820 0x7f5cd010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:30.269 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 >> 172.21.15.71:0/1335801178 conn(0x7f5cd00f0ce0 msgr2=0x7f5cd00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:30.269 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 shutdown_connections 2021-11-20T07:42:30.270 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:30.263+0000 7f5cd837b700 1 -- 172.21.15.71:0/1335801178 wait complete. 2021-11-20T07:42:30.271 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:30.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:30.277 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:31.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:30 smithi112 conmon[36062]: audit 2021-11-20T07:42:30.262907+0000 mon.smithi071 (mon.0) 853 : audit [DBG] 2021-11-20T07:42:31.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:30 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/1335801178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:31.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:30 smithi112 conmon[36062]: cluster 2021-11-20T07:42:30.422006+0000 mgr.smithi071.ynqmwn (mgr.14162) 316 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:31.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:30 smithi071 conmon[33687]: audit 2021-11-20T07:42:30.262907+0000 mon.smithi071 (mon.0) 853 2021-11-20T07:42:31.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:30 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/1335801178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:31.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:30 smithi071 conmon[33687]: cluster 2021-11-20T07:42:30.422006+0000 mgr.smithi071.ynqmwn 2021-11-20T07:42:31.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:30 smithi071 conmon[33687]: (mgr.14162) 316 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:31.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.402+0000 7f282d097700 1 -- 172.21.15.71:0/1927094854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f7920 msgr2=0x7f28280f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.402+0000 7f282d097700 1 --2- 172.21.15.71:0/1927094854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f7920 0x7f28280f7d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f2810004660 tx=0x7f2810009b30).stop 2021-11-20T07:42:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- 172.21.15.71:0/1927094854 shutdown_connections 2021-11-20T07:42:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 --2- 172.21.15.71:0/1927094854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f7920 0x7f28280f7d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 --2- 172.21.15.71:0/1927094854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f6680 0x7f28280f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- 172.21.15.71:0/1927094854 >> 172.21.15.71:0/1927094854 conn(0x7f28280f1de0 msgr2=0x7f28280f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:31.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- 172.21.15.71:0/1927094854 shutdown_connections 2021-11-20T07:42:31.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- 172.21.15.71:0/1927094854 wait complete. 2021-11-20T07:42:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 Processor -- start 2021-11-20T07:42:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- start start 2021-11-20T07:42:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f7920 0x7f2828107110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828107710 con 0x7f28280f6680 2021-11-20T07:42:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.403+0000 7f282d097700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828107850 con 0x7f28280f7920 2021-11-20T07:42:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49510/0 (socket says 172.21.15.71:49510) 2021-11-20T07:42:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 -- 172.21.15.71:0/1172203676 learned_addr learned my addr 172.21.15.71:0/1172203676 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f282659c700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f7920 0x7f2828107110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 -- 172.21.15.71:0/1172203676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f7920 msgr2=0x7f2828107110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f7920 0x7f2828107110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 -- 172.21.15.71:0/1172203676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810005040 con 0x7f28280f6680 2021-11-20T07:42:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f2826d9d700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f281800d7f0 tx=0x7f2818004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f281800b4f0 con 0x7f28280f6680 2021-11-20T07:42:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282810c260 con 0x7f28280f6680 2021-11-20T07:42:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f281800f040 con 0x7f28280f6680 2021-11-20T07:42:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2818013980 con 0x7f28280f6680 2021-11-20T07:42:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.404+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282810c820 con 0x7f28280f6680 2021-11-20T07:42:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.405+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2828027a00 con 0x7f28280f6680 2021-11-20T07:42:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.405+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f281801a030 con 0x7f28280f6680 2021-11-20T07:42:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.406+0000 7f281f7fe700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 0x7f281405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.406+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2818076290 con 0x7f28280f6680 2021-11-20T07:42:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.406+0000 7f282659c700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 0x7f281405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.406+0000 7f282659c700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 0x7f281405e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2810009d60 tx=0x7f2810009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.410+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f281804c480 con 0x7f28280f6680 2021-11-20T07:42:31.612 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.610+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28280fee10 con 0x7f28280f6680 2021-11-20T07:42:31.613 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.611+0000 7f281f7fe700 1 -- 172.21.15.71:0/1172203676 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f2818018090 con 0x7f28280f6680 2021-11-20T07:42:31.613 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:31.613 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:31.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 msgr2=0x7f281405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:31.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 0x7f281405e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2810009d60 tx=0x7f2810009b30).stop 2021-11-20T07:42:31.615 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 msgr2=0x7f2828106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:31.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f281800d7f0 tx=0x7f2818004e80).stop 2021-11-20T07:42:31.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 shutdown_connections 2021-11-20T07:42:31.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f281405c2a0 0x7f281405e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28280f6680 0x7f2828106c20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.616 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 --2- 172.21.15.71:0/1172203676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28280f7920 0x7f2828107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:31.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 >> 172.21.15.71:0/1172203676 conn(0x7f28280f1de0 msgr2=0x7f28280fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:31.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.612+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 shutdown_connections 2021-11-20T07:42:31.617 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:31.613+0000 7f282d097700 1 -- 172.21.15.71:0/1172203676 wait complete. 2021-11-20T07:42:31.617 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:31.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:31.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:32.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:32.762 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- 172.21.15.71:0/191851953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f86a0 msgr2=0x7fbe640f4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:32.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 --2- 172.21.15.71:0/191851953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f86a0 0x7fbe640f4820 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fbe54004660 tx=0x7fbe54009b30).stop 2021-11-20T07:42:32.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- 172.21.15.71:0/191851953 shutdown_connections 2021-11-20T07:42:32.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 --2- 172.21.15.71:0/191851953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f86a0 0x7fbe640f4820 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:32.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 --2- 172.21.15.71:0/191851953 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe640f7d00 0x7fbe640f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:32.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- 172.21.15.71:0/191851953 >> 172.21.15.71:0/191851953 conn(0x7fbe640f0440 msgr2=0x7fbe640f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:32.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- 172.21.15.71:0/191851953 shutdown_connections 2021-11-20T07:42:32.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- 172.21.15.71:0/191851953 wait complete. 2021-11-20T07:42:32.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 Processor -- start 2021-11-20T07:42:32.764 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- start start 2021-11-20T07:42:32.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:32.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe640f86a0 0x7fbe641073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:32.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64107950 con 0x7fbe640f7d00 2021-11-20T07:42:32.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe6b895700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64107a90 con 0x7fbe640f86a0 2021-11-20T07:42:32.765 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe69631700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:32.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe69631700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49520/0 (socket says 172.21.15.71:49520) 2021-11-20T07:42:32.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.755+0000 7fbe69631700 1 -- 172.21.15.71:0/390491690 learned_addr learned my addr 172.21.15.71:0/390491690 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:32.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe69631700 1 -- 172.21.15.71:0/390491690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe640f86a0 msgr2=0x7fbe641073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:32.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe69631700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe640f86a0 0x7fbe641073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:32.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe69631700 1 -- 172.21.15.71:0/390491690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54005040 con 0x7fbe640f7d00 2021-11-20T07:42:32.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe69631700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fbe60002700 tx=0x7fbe60009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:32.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60009320 con 0x7fbe640f7d00 2021-11-20T07:42:32.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe6000f040 con 0x7fbe640f7d00 2021-11-20T07:42:32.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60014900 con 0x7fbe640f7d00 2021-11-20T07:42:32.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6410c530 con 0x7fbe640f7d00 2021-11-20T07:42:32.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.756+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6410caf0 con 0x7fbe640f7d00 2021-11-20T07:42:32.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.757+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64027a00 con 0x7fbe640f7d00 2021-11-20T07:42:32.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.757+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbe60009480 con 0x7fbe640f7d00 2021-11-20T07:42:32.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.758+0000 7fbe59ffb700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 0x7fbe5005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:32.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.758+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fbe60076720 con 0x7fbe640f7d00 2021-11-20T07:42:32.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.758+0000 7fbe68e30700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 0x7fbe5005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:32.769 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.758+0000 7fbe68e30700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 0x7fbe5005e760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fbe54009f50 tx=0x7fbe54009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:32.770 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.762+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe60048ee0 con 0x7fbe640f7d00 2021-11-20T07:42:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:32 smithi112 conmon[36062]: audit 2021-11-20T07:42:31.612191+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:32.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:32 smithi112 conmon[36062]: 854 : audit [DBG] from='client.? 172.21.15.71:0/1172203676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:32.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:32 smithi071 conmon[33687]: audit 2021-11-20T07:42:31.612191+0000 mon.smithi071 (mon.0) 854 : audit [DBG] from='client.? 172.21.15.71:0/1172203676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:32.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.946+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe640fee40 con 0x7fbe640f7d00 2021-11-20T07:42:32.949 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.947+0000 7fbe59ffb700 1 -- 172.21.15.71:0/390491690 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7fbe6004c500 con 0x7fbe640f7d00 2021-11-20T07:42:32.950 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:32.950 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 msgr2=0x7fbe5005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 0x7fbe5005e760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fbe54009f50 tx=0x7fbe54009b30).stop 2021-11-20T07:42:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 msgr2=0x7fbe64106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:32.952 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fbe60002700 tx=0x7fbe60009d80).stop 2021-11-20T07:42:32.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 shutdown_connections 2021-11-20T07:42:32.953 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fbe5005c2a0 0x7fbe5005e760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:32.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe640f7d00 0x7fbe64106eb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:32.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 --2- 172.21.15.71:0/390491690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe640f86a0 0x7fbe641073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:32.954 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.948+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 >> 172.21.15.71:0/390491690 conn(0x7fbe640f0440 msgr2=0x7fbe640fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:32.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.949+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 shutdown_connections 2021-11-20T07:42:32.955 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:32.949+0000 7fbe6b895700 1 -- 172.21.15.71:0/390491690 wait complete. 2021-11-20T07:42:32.955 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:32.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:32.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:33.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:33 smithi112 conmon[36062]: cluster 2021-11-20T07:42:32.422440+0000 mgr.smithi071.ynqmwn (mgr.14162) 317 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:33.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:33 smithi112 conmon[36062]: audit 2021-11-20T07:42:32.948302+0000 mon.smithi071 (mon.0) 855 : audit [DBG] from='client.? 172.21.15.71:0/390491690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:33 smithi071 conmon[33687]: cluster 2021-11-20T07:42:32.422440+0000 mgr.smithi071.ynqmwn (mgr.14162) 317 2021-11-20T07:42:33.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:33 smithi071 conmon[33687]: : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:33.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:33 smithi071 conmon[33687]: audit 2021-11-20T07:42:32.948302+0000 mon.smithi071 (mon.0) 855 : audit [DBG] from='client.? 172.21.15.71:0/390491690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:33.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 -- 172.21.15.71:0/2125911004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 msgr2=0x7f01ac0f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/2125911004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac0f4d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f019c004660 tx=0x7f019c009b30).stop 2021-11-20T07:42:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 -- 172.21.15.71:0/2125911004 shutdown_connections 2021-11-20T07:42:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/2125911004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac0f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/2125911004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac0f4d80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 -- 172.21.15.71:0/2125911004 >> 172.21.15.71:0/2125911004 conn(0x7f01ac0f0440 msgr2=0x7f01ac0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:34.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 -- 172.21.15.71:0/2125911004 shutdown_connections 2021-11-20T07:42:34.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.089+0000 7f01b0e5c700 1 -- 172.21.15.71:0/2125911004 wait complete. 2021-11-20T07:42:34.095 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 Processor -- start 2021-11-20T07:42:34.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 -- start start 2021-11-20T07:42:34.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:34.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac1073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:34.096 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac107990 con 0x7f01ac0f4960 2021-11-20T07:42:34.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac107ad0 con 0x7f01ac0f5300 2021-11-20T07:42:34.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:34.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49530/0 (socket says 172.21.15.71:49530) 2021-11-20T07:42:34.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 -- 172.21.15.71:0/3343443201 learned_addr learned my addr 172.21.15.71:0/3343443201 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01a9d9b700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac1073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 -- 172.21.15.71:0/3343443201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 msgr2=0x7f01ac1073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac1073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 -- 172.21.15.71:0/3343443201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c005040 con 0x7f01ac0f4960 2021-11-20T07:42:34.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01a9d9b700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac1073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:34.103 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01aa59c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f019c004e30 tx=0x7f019c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:34.103 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019c018070 con 0x7f01ac0f4960 2021-11-20T07:42:34.104 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f019c013410 con 0x7f01ac0f4960 2021-11-20T07:42:34.104 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.090+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac10c510 con 0x7f01ac0f4960 2021-11-20T07:42:34.104 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.091+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019c01c3c0 con 0x7f01ac0f4960 2021-11-20T07:42:34.105 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.091+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac10cad0 con 0x7f01ac0f4960 2021-11-20T07:42:34.105 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.091+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac027a00 con 0x7f01ac0f4960 2021-11-20T07:42:34.106 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.092+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f019c01d420 con 0x7f01ac0f4960 2021-11-20T07:42:34.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.092+0000 7f019affd700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 0x7f019405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:34.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.092+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f019c077fc0 con 0x7f01ac0f4960 2021-11-20T07:42:34.107 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.092+0000 7f01a9d9b700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 0x7f019405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:34.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.092+0000 7f01a9d9b700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 0x7f019405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f01ac0f5740 tx=0x7f01a0009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:34.108 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.096+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f019c04a780 con 0x7f01ac0f4960 2021-11-20T07:42:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.285+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01ac0fee70 con 0x7f01ac0f4960 2021-11-20T07:42:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.286+0000 7f019affd700 1 -- 172.21.15.71:0/3343443201 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f019c04dda0 con 0x7f01ac0f4960 2021-11-20T07:42:34.291 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:34.291 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 msgr2=0x7f019405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 0x7f019405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f01ac0f5740 tx=0x7f01a0009be0).stop 2021-11-20T07:42:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 msgr2=0x7f01ac106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f019c004e30 tx=0x7f019c009b30).stop 2021-11-20T07:42:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 shutdown_connections 2021-11-20T07:42:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f019405c2a0 0x7f019405e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01ac0f4960 0x7f01ac106ef0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.287+0000 7f01b0e5c700 1 --2- 172.21.15.71:0/3343443201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01ac0f5300 0x7f01ac1073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.288+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 >> 172.21.15.71:0/3343443201 conn(0x7f01ac0f0440 msgr2=0x7f01ac0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.288+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 shutdown_connections 2021-11-20T07:42:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:34.288+0000 7f01b0e5c700 1 -- 172.21.15.71:0/3343443201 wait complete. 2021-11-20T07:42:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:34.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:34.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:34.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:34 smithi071 conmon[33687]: audit 2021-11-20T07:42:34.287076+0000 mon.smithi071 (mon. 2021-11-20T07:42:34.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:34 smithi071 conmon[33687]: 0) 856 : audit [DBG] from='client.? 172.21.15.71:0/3343443201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:34.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:34 smithi112 conmon[36062]: audit 2021-11-20T07:42:34.287076+0000 mon.smithi071 (mon.0) 856 : audit 2021-11-20T07:42:34.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:34 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/3343443201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:35.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:35.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 -- 172.21.15.71:0/3030811761 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 msgr2=0x7f97240f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:35.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 --2- 172.21.15.71:0/3030811761 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f97240f59c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9714004660 tx=0x7f9714009b30).stop 2021-11-20T07:42:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 -- 172.21.15.71:0/3030811761 shutdown_connections 2021-11-20T07:42:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 --2- 172.21.15.71:0/3030811761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f97240f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 --2- 172.21.15.71:0/3030811761 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f97240f59c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 -- 172.21.15.71:0/3030811761 >> 172.21.15.71:0/3030811761 conn(0x7f97240f0ce0 msgr2=0x7f97240f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:35.436 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 -- 172.21.15.71:0/3030811761 shutdown_connections 2021-11-20T07:42:35.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 -- 172.21.15.71:0/3030811761 wait complete. 2021-11-20T07:42:35.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.431+0000 7f972a18d700 1 Processor -- start 2021-11-20T07:42:35.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f972a18d700 1 -- start start 2021-11-20T07:42:35.437 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f972a18d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:35.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f972a18d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f972410bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:35.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f972a18d700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972410c1d0 con 0x7f97240f55a0 2021-11-20T07:42:35.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f972a18d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972410c310 con 0x7f97240f6840 2021-11-20T07:42:35.438 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:35.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49542/0 (socket says 172.21.15.71:49542) 2021-11-20T07:42:35.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 -- 172.21.15.71:0/344627653 learned_addr learned my addr 172.21.15.71:0/344627653 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:35.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f9722ffd700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f972410bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:35.439 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 -- 172.21.15.71:0/344627653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 msgr2=0x7f972410bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:35.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f972410bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 -- 172.21.15.71:0/344627653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f97240f55a0 2021-11-20T07:42:35.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f9722ffd700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f972410bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:35.440 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.432+0000 7f97237fe700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9714009f50 tx=0x7f9714009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:35.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9714018070 con 0x7f97240f55a0 2021-11-20T07:42:35.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972410c510 con 0x7f97240f55a0 2021-11-20T07:42:35.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9714013410 con 0x7f97240f55a0 2021-11-20T07:42:35.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971401c620 con 0x7f97240f55a0 2021-11-20T07:42:35.441 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972410ca40 con 0x7f97240f55a0 2021-11-20T07:42:35.442 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.433+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724027a00 con 0x7f97240f55a0 2021-11-20T07:42:35.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.434+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f971401d420 con 0x7f97240f55a0 2021-11-20T07:42:35.444 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.434+0000 7f970bfff700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 0x7f970c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:35.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.434+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f9714078230 con 0x7f97240f55a0 2021-11-20T07:42:35.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.434+0000 7f9722ffd700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 0x7f970c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:35.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.435+0000 7f9722ffd700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 0x7f970c05e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f97180041d0 tx=0x7f9718009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:35.445 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.438+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f971404a9f0 con 0x7f97240f55a0 2021-11-20T07:42:35.631 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.629+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97240586e0 con 0x7f97240f55a0 2021-11-20T07:42:35.631 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.629+0000 7f970bfff700 1 -- 172.21.15.71:0/344627653 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f971404e010 con 0x7f97240f55a0 2021-11-20T07:42:35.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:35.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 msgr2=0x7f970c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:35.634 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 0x7f970c05e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f97180041d0 tx=0x7f9718009040).stop 2021-11-20T07:42:35.634 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 msgr2=0x7f972405ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:35.634 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9714009f50 tx=0x7f9714009b30).stop 2021-11-20T07:42:35.634 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 shutdown_connections 2021-11-20T07:42:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f970c05c2a0 0x7f970c05e760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97240f55a0 0x7f972405ff00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 --2- 172.21.15.71:0/344627653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97240f6840 0x7f972410bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 >> 172.21.15.71:0/344627653 conn(0x7f97240f0ce0 msgr2=0x7f97240fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 shutdown_connections 2021-11-20T07:42:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:35.631+0000 7f972a18d700 1 -- 172.21.15.71:0/344627653 wait complete. 2021-11-20T07:42:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:35.646 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:35 smithi071 conmon[33687]: cluster 2021-11-20T07:42:34.422831+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:42:35.646 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:35 smithi071 conmon[33687]: ) 318 : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:35.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:35.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:35 smithi112 conmon[36062]: cluster 2021-11-20T07:42:34.422831+0000 mgr.smithi071.ynqmwn (mgr.14162) 318 2021-11-20T07:42:35.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:35 smithi112 conmon[36062]: : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T07:42:36.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:36.680 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:36 smithi071 conmon[33687]: audit 2021-11-20T07:42:35.518275+0000 mon.smithi071 (mon.0) 857 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:42:36.680 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:36 smithi071 conmon[33687]: audit 2021-11-20T07:42:35.630789+0000 mon.smithi071 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.71:0/344627653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:36.680 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:36 smithi071 conmon[33687]: audit 2021-11-20T07:42:35.889730+0000 mon.smithi071 (mon.0) 859 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:42:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- 172.21.15.71:0/2436285863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 msgr2=0x7f65ac0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 --2- 172.21.15.71:0/2436285863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f58f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f65a8004660 tx=0x7f65a8009b30).stop 2021-11-20T07:42:36.780 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- 172.21.15.71:0/2436285863 shutdown_connections 2021-11-20T07:42:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 --2- 172.21.15.71:0/2436285863 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 0x7f65ac0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 --2- 172.21.15.71:0/2436285863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f58f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- 172.21.15.71:0/2436285863 >> 172.21.15.71:0/2436285863 conn(0x7f65ac0f0c10 msgr2=0x7f65ac0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- 172.21.15.71:0/2436285863 shutdown_connections 2021-11-20T07:42:36.781 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- 172.21.15.71:0/2436285863 wait complete. 2021-11-20T07:42:36.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 Processor -- start 2021-11-20T07:42:36.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- start start 2021-11-20T07:42:36.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:36.782 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 0x7f65ac0f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:36.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0f80a0 con 0x7f65ac0f54d0 2021-11-20T07:42:36.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b3293700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0f81e0 con 0x7f65ac0f6770 2021-11-20T07:42:36.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b2291700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:36.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b2291700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49552/0 (socket says 172.21.15.71:49552) 2021-11-20T07:42:36.783 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.776+0000 7f65b2291700 1 -- 172.21.15.71:0/2967037345 learned_addr learned my addr 172.21.15.71:0/2967037345 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:36.784 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b1a90700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 0x7f65ac0f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:36.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b2291700 1 -- 172.21.15.71:0/2967037345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 msgr2=0x7f65ac0f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:36.785 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b2291700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 0x7f65ac0f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:36.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b2291700 1 -- 172.21.15.71:0/2967037345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8005040 con 0x7f65ac0f54d0 2021-11-20T07:42:36.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b2291700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f65a8009f50 tx=0x7f65a8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:36.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a8018070 con 0x7f65ac0f54d0 2021-11-20T07:42:36.786 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac0f8460 con 0x7f65ac0f54d0 2021-11-20T07:42:36.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65a8013410 con 0x7f65ac0f54d0 2021-11-20T07:42:36.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a801c620 con 0x7f65ac0f54d0 2021-11-20T07:42:36.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.777+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac10ed80 con 0x7f65ac0f54d0 2021-11-20T07:42:36.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.778+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac045ad0 con 0x7f65ac0f54d0 2021-11-20T07:42:36.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.778+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f65a801d420 con 0x7f65ac0f54d0 2021-11-20T07:42:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:36 smithi112 conmon[36062]: audit 2021-11-20T07:42:35.518275+0000 mon.smithi071 (mon.0) 857 : 2021-11-20T07:42:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:36 smithi112 conmon[36062]: audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T07:42:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:36 smithi112 conmon[36062]: audit 2021-11-20T07:42:35.630789+0000 mon.smithi071 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.71:0/344627653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:36 smithi112 conmon[36062]: audit 2021-11-20T07:42:35.889730+0000 mon.smithi071 (mon.0) 859 : audit [INF] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' 2021-11-20T07:42:36.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.779+0000 7f65a2ffd700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 0x7f659805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:36.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.779+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f65a8078240 con 0x7f65ac0f54d0 2021-11-20T07:42:36.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.779+0000 7f65b1a90700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 0x7f659805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:36.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.779+0000 7f65b1a90700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 0x7f659805e7c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f659c00b830 tx=0x7f659c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:36.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.782+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65a804aa00 con 0x7f65ac0f54d0 2021-11-20T07:42:36.967 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.966+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65ac0fdc70 con 0x7f65ac0f54d0 2021-11-20T07:42:36.968 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.966+0000 7f65a2ffd700 1 -- 172.21.15.71:0/2967037345 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f65a804e020 con 0x7f65ac0f54d0 2021-11-20T07:42:36.971 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:36.971 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:36.972 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 msgr2=0x7f659805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:36.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 0x7f659805e7c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f659c00b830 tx=0x7f659c008040).stop 2021-11-20T07:42:36.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 msgr2=0x7f65ac0f9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:36.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f65a8009f50 tx=0x7f65a8009b30).stop 2021-11-20T07:42:36.973 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 shutdown_connections 2021-11-20T07:42:36.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f659805c300 0x7f659805e7c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:36.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65ac0f54d0 0x7f65ac0f9630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:36.974 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 --2- 172.21.15.71:0/2967037345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65ac0f6770 0x7f65ac0f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:36.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 >> 172.21.15.71:0/2967037345 conn(0x7f65ac0f0c10 msgr2=0x7f65ac05ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:36.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 shutdown_connections 2021-11-20T07:42:36.975 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:36.968+0000 7f65b3293700 1 -- 172.21.15.71:0/2967037345 wait complete. 2021-11-20T07:42:36.976 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:36.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:36.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:37 smithi112 conmon[36062]: cluster 2021-11-20T07:42:36.423225+0000 mgr.smithi071.ynqmwn (mgr.14162) 319 : 2021-11-20T07:42:37.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:37 smithi112 conmon[36062]: cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:42:37.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:37 smithi112 conmon[36062]: audit 2021-11-20T07:42:36.967579+0000 mon.smithi071 (mon.0) 860 : audit [DBG] from='client.? 172.21.15.71:0/2967037345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:37.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:37 smithi071 conmon[33687]: cluster 2021-11-20T07:42:36.423225+0000 mgr.smithi071.ynqmwn (mgr.14162 2021-11-20T07:42:37.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:37 smithi071 conmon[33687]: ) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:42:37.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:37 smithi071 conmon[33687]: audit 2021-11-20T07:42:36.967579+0000 mon.smithi071 (mon.0) 860 : audit [DBG] from='client.? 172.21.15.71:0/2967037345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:37.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:38.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/3785290891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 msgr2=0x7f0ff40f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:38.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/3785290891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff40f4d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0fe8004660 tx=0x7f0fe8009b30).stop 2021-11-20T07:42:38.117 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/3785290891 shutdown_connections 2021-11-20T07:42:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/3785290891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ff40f5300 0x7f0ff40f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/3785290891 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff40f4d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/3785290891 >> 172.21.15.71:0/3785290891 conn(0x7f0ff40f0440 msgr2=0x7f0ff40f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/3785290891 shutdown_connections 2021-11-20T07:42:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.111+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/3785290891 wait complete. 2021-11-20T07:42:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 Processor -- start 2021-11-20T07:42:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 -- start start 2021-11-20T07:42:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:38.119 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ff40f5300 0x7f0ff41073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4107990 con 0x7f0ff40f4960 2021-11-20T07:42:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff9fd6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4107ad0 con 0x7f0ff40f5300 2021-11-20T07:42:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff37fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49562/0 (socket says 172.21.15.71:49562) 2021-11-20T07:42:38.120 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.112+0000 7f0ff37fe700 1 -- 172.21.15.71:0/2734945220 learned_addr learned my addr 172.21.15.71:0/2734945220 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff37fe700 1 -- 172.21.15.71:0/2734945220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ff40f5300 msgr2=0x7f0ff41073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff37fe700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ff40f5300 0x7f0ff41073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff37fe700 1 -- 172.21.15.71:0/2734945220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8005040 con 0x7f0ff40f4960 2021-11-20T07:42:38.121 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff37fe700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0fe8004e30 tx=0x7f0fe8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:38.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe8018070 con 0x7f0ff40f4960 2021-11-20T07:42:38.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff410c510 con 0x7f0ff40f4960 2021-11-20T07:42:38.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fe8013410 con 0x7f0ff40f4960 2021-11-20T07:42:38.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe801c3c0 con 0x7f0ff40f4960 2021-11-20T07:42:38.122 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.113+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff410cad0 con 0x7f0ff40f4960 2021-11-20T07:42:38.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.114+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff4027a00 con 0x7f0ff40f4960 2021-11-20T07:42:38.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.114+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0fe801d420 con 0x7f0ff40f4960 2021-11-20T07:42:38.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.114+0000 7f0fdbfff700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 0x7f0fdc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:38.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.115+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f0fe8078060 con 0x7f0ff40f4960 2021-11-20T07:42:38.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.115+0000 7f0ff2ffd700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 0x7f0fdc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:38.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.115+0000 7f0ff2ffd700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 0x7f0fdc05e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0ff40f5740 tx=0x7f0fe40099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:38.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.118+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fe804a820 con 0x7f0ff40f4960 2021-11-20T07:42:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.302+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff40fee70 con 0x7f0ff40f4960 2021-11-20T07:42:38.304 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.302+0000 7f0fdbfff700 1 -- 172.21.15.71:0/2734945220 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f0fe804de40 con 0x7f0ff40f4960 2021-11-20T07:42:38.304 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:38.305 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 msgr2=0x7f0fdc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 0x7f0fdc05e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0ff40f5740 tx=0x7f0fe40099e0).stop 2021-11-20T07:42:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 msgr2=0x7f0ff4106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0fe8004e30 tx=0x7f0fe8009b30).stop 2021-11-20T07:42:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 shutdown_connections 2021-11-20T07:42:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f0fdc05c2a0 0x7f0fdc05e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ff40f4960 0x7f0ff4106ef0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 --2- 172.21.15.71:0/2734945220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ff40f5300 0x7f0ff41073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 >> 172.21.15.71:0/2734945220 conn(0x7f0ff40f0440 msgr2=0x7f0ff40fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 shutdown_connections 2021-11-20T07:42:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:38.304+0000 7f0ff9fd6700 1 -- 172.21.15.71:0/2734945220 wait complete. 2021-11-20T07:42:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:38.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:38.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:38.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:38 smithi071 conmon[33687]: audit 2021-11-20T07:42:38.303462+0000 mon.smithi071 (mon.0) 861 : audit [DBG] from='client.? 172.21.15.71:0/2734945220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:38.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:38 smithi112 conmon[36062]: audit 2021-11-20T07:42:38.303462+0000 mon.smithi071 (mon.0) 861 : audit [DBG] from='client.? 172.21.15.71:0/2734945220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:39.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:39.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 -- 172.21.15.71:0/1388876786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 msgr2=0x7f65cc0f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:39.449 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 --2- 172.21.15.71:0/1388876786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc0f7090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f65bc004660 tx=0x7f65bc009b30).stop 2021-11-20T07:42:39.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 -- 172.21.15.71:0/1388876786 shutdown_connections 2021-11-20T07:42:39.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 --2- 172.21.15.71:0/1388876786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 0x7f65cc0f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 --2- 172.21.15.71:0/1388876786 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc0f7090 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.450 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 -- 172.21.15.71:0/1388876786 >> 172.21.15.71:0/1388876786 conn(0x7f65cc0f04a0 msgr2=0x7f65cc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:39.451 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 -- 172.21.15.71:0/1388876786 shutdown_connections 2021-11-20T07:42:39.451 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.445+0000 7f65d2452700 1 -- 172.21.15.71:0/1388876786 wait complete. 2021-11-20T07:42:39.451 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 Processor -- start 2021-11-20T07:42:39.451 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 -- start start 2021-11-20T07:42:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 0x7f65cc0f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc107b60 con 0x7f65cc0f6c70 2021-11-20T07:42:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65d2452700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc10e670 con 0x7f65cc0f5970 2021-11-20T07:42:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49574/0 (socket says 172.21.15.71:49574) 2021-11-20T07:42:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cb7fe700 1 -- 172.21.15.71:0/4001882107 learned_addr learned my addr 172.21.15.71:0/4001882107 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cbfff700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 0x7f65cc0f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cb7fe700 1 -- 172.21.15.71:0/4001882107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 msgr2=0x7f65cc0f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.446+0000 7f65cb7fe700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 0x7f65cc0f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65cb7fe700 1 -- 172.21.15.71:0/4001882107 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc005040 con 0x7f65cc0f6c70 2021-11-20T07:42:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65cb7fe700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f65c00026d0 tx=0x7f65c0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c00079c0 con 0x7f65cc0f6c70 2021-11-20T07:42:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65c000f040 con 0x7f65cc0f6c70 2021-11-20T07:42:39.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c00148b0 con 0x7f65cc0f6c70 2021-11-20T07:42:39.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc10e810 con 0x7f65cc0f6c70 2021-11-20T07:42:39.455 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.447+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc10eda0 con 0x7f65cc0f6c70 2021-11-20T07:42:39.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.448+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc046330 con 0x7f65cc0f6c70 2021-11-20T07:42:39.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.448+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f65c0007b20 con 0x7f65cc0f6c70 2021-11-20T07:42:39.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.449+0000 7f65c8ff9700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 0x7f65b405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:39.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.449+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f65c0076340 con 0x7f65cc0f6c70 2021-11-20T07:42:39.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.449+0000 7f65cbfff700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 0x7f65b405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:39.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.449+0000 7f65cbfff700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 0x7f65b405e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f65bc005010 tx=0x7f65bc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:39.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.452+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65c0048b00 con 0x7f65cc0f6c70 2021-11-20T07:42:39.640 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.638+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65cc027a00 con 0x7f65cc0f6c70 2021-11-20T07:42:39.640 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.638+0000 7f65c8ff9700 1 -- 172.21.15.71:0/4001882107 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f65c004c120 con 0x7f65cc0f6c70 2021-11-20T07:42:39.641 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:39.642 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:39.644 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.639+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 msgr2=0x7f65b405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:39.645 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 0x7f65b405e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f65bc005010 tx=0x7f65bc009b30).stop 2021-11-20T07:42:39.645 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 msgr2=0x7f65cc10e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:39.646 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f65c00026d0 tx=0x7f65c0009de0).stop 2021-11-20T07:42:39.646 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 shutdown_connections 2021-11-20T07:42:39.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f65cc0f5970 0x7f65cc0f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.647 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f65b405c2a0 0x7f65b405e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 --2- 172.21.15.71:0/4001882107 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65cc0f6c70 0x7f65cc10e180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:39.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 >> 172.21.15.71:0/4001882107 conn(0x7f65cc0f04a0 msgr2=0x7f65cc0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:39.648 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 shutdown_connections 2021-11-20T07:42:39.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:39.640+0000 7f65d2452700 1 -- 172.21.15.71:0/4001882107 wait complete. 2021-11-20T07:42:39.649 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:39.653 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:39 smithi071 conmon[33687]: cluster 2021-11-20T07:42:38.423818+0000 mgr.smithi071.ynqmwn (mgr.14162) 320 : cluster 2021-11-20T07:42:39.653 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:39 smithi071 conmon[33687]: [DBG] pgmap v296: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T07:42:39.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:39.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:39.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:39 smithi112 conmon[36062]: cluster 2021-11-20T07:42:38.423818+0000 mgr.smithi071.ynqmwn (mgr 2021-11-20T07:42:39.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:39 smithi112 conmon[36062]: .14162) 320 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T07:42:40.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:40 smithi112 conmon[36062]: audit 2021-11-20T07:42:39.639498+0000 mon.smithi071 (mon.0) 862 : audit [DBG] from='client.? 172.21.15.71:0/4001882107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:40.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:40 smithi112 conmon[36062]: cluster 2021-11-20T07:42:40.424181+0000 mgr.smithi071.ynqmwn (mgr.14162) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:42:40.787 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 -- 172.21.15.71:0/3356154541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 msgr2=0x7f66b40f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 --2- 172.21.15.71:0/3356154541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b40f69f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f66a4004660 tx=0x7f66a4009b30).stop 2021-11-20T07:42:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 -- 172.21.15.71:0/3356154541 shutdown_connections 2021-11-20T07:42:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 --2- 172.21.15.71:0/3356154541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 0x7f66b40f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 --2- 172.21.15.71:0/3356154541 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b40f69f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 -- 172.21.15.71:0/3356154541 >> 172.21.15.71:0/3356154541 conn(0x7f66b40f1d10 msgr2=0x7f66b40f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 -- 172.21.15.71:0/3356154541 shutdown_connections 2021-11-20T07:42:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.784+0000 7f66b9379700 1 -- 172.21.15.71:0/3356154541 wait complete. 2021-11-20T07:42:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 Processor -- start 2021-11-20T07:42:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 -- start start 2021-11-20T07:42:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 0x7f66b4107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4107890 con 0x7f66b40f65d0 2021-11-20T07:42:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b9379700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b41080d0 con 0x7f66b40f7870 2021-11-20T07:42:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49582/0 (socket says 172.21.15.71:49582) 2021-11-20T07:42:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.785+0000 7f66b3fff700 1 -- 172.21.15.71:0/2049878425 learned_addr learned my addr 172.21.15.71:0/2049878425 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b37fe700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 0x7f66b4107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b3fff700 1 -- 172.21.15.71:0/2049878425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 msgr2=0x7f66b4107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b3fff700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 0x7f66b4107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b3fff700 1 -- 172.21.15.71:0/2049878425 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4005040 con 0x7f66b40f65d0 2021-11-20T07:42:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b3fff700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f66a4004e30 tx=0x7f66a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:40.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a4018070 con 0x7f66b40f65d0 2021-11-20T07:42:40.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b4108350 con 0x7f66b40f65d0 2021-11-20T07:42:40.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a4013410 con 0x7f66b40f65d0 2021-11-20T07:42:40.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a401c3c0 con 0x7f66b40f65d0 2021-11-20T07:42:40.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.786+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b4108830 con 0x7f66b40f65d0 2021-11-20T07:42:40.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.787+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b40f59c0 con 0x7f66b40f65d0 2021-11-20T07:42:40.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.787+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f66a401d420 con 0x7f66b40f65d0 2021-11-20T07:42:40.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.787+0000 7f66b0ff9700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 0x7f669c05e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:40.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.788+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f66a40780b0 con 0x7f66b40f65d0 2021-11-20T07:42:40.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.788+0000 7f66b37fe700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 0x7f669c05e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:40.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.788+0000 7f66b37fe700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 0x7f669c05e810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f66b4106b30 tx=0x7f66a8009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:40.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.791+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66a404a7d0 con 0x7f66b40f65d0 2021-11-20T07:42:40.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:40 smithi071 conmon[33687]: audit 2021-11-20T07:42:39.639498+0000 mon.smithi071 (mon.0) 862 : audit [DBG] from='client.? 172.21.15.71:0/4001882107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:40.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:40 smithi071 conmon[33687]: 2021-11-20T07:42:40.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:40 smithi071 conmon[33687]: cluster 2021-11-20T07:42:40.424181+0000 mgr.smithi071.ynqmwn (mgr.14162) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:42:40.976 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.975+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b4027b20 con 0x7f66b40f65d0 2021-11-20T07:42:40.976 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.975+0000 7f66b0ff9700 1 -- 172.21.15.71:0/2049878425 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f66a404ddf0 con 0x7f66b40f65d0 2021-11-20T07:42:40.977 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:40.978 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:40.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 msgr2=0x7f669c05e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:40.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 0x7f669c05e810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f66b4106b30 tx=0x7f66a8009be0).stop 2021-11-20T07:42:40.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 msgr2=0x7f66b4106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:40.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f66a4004e30 tx=0x7f66a4009b30).stop 2021-11-20T07:42:40.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 shutdown_connections 2021-11-20T07:42:40.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f669c05c350 0x7f669c05e810 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b40f65d0 0x7f66b4106da0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 --2- 172.21.15.71:0/2049878425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66b40f7870 0x7f66b4107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:40.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 >> 172.21.15.71:0/2049878425 conn(0x7f66b40f1d10 msgr2=0x7f66b40fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:40.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 shutdown_connections 2021-11-20T07:42:40.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:40.977+0000 7f66b9379700 1 -- 172.21.15.71:0/2049878425 wait complete. 2021-11-20T07:42:40.984 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:40.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:40.991 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:41.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:41 smithi112 conmon[36062]: audit 2021-11-20T07:42:40.975865+0000 mon.smithi071 (mon.0) 863 2021-11-20T07:42:41.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:41 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/2049878425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:41.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:41 smithi071 conmon[33687]: audit 2021-11-20T07:42:40.975865+0000 mon.smithi071 (mon.0) 863 : audit [DBG] from='client.? 172.21.15.71:0/2049878425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:41.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:42.123 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 -- 172.21.15.71:0/1070202995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 msgr2=0x7f75cc0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:42.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/1070202995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f4c60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f75bc004660 tx=0x7f75bc009b30).stop 2021-11-20T07:42:42.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 -- 172.21.15.71:0/1070202995 shutdown_connections 2021-11-20T07:42:42.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/1070202995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75cc0f51e0 0x7f75cc0f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/1070202995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f4c60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 -- 172.21.15.71:0/1070202995 >> 172.21.15.71:0/1070202995 conn(0x7f75cc0f0360 msgr2=0x7f75cc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 -- 172.21.15.71:0/1070202995 shutdown_connections 2021-11-20T07:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.118+0000 7f75d0ac3700 1 -- 172.21.15.71:0/1070202995 wait complete. 2021-11-20T07:42:42.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 Processor -- start 2021-11-20T07:42:42.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 -- start start 2021-11-20T07:42:42.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:42.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75cc0f51e0 0x7f75cc0f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:42.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0f5dd0 con 0x7f75cc0f4840 2021-11-20T07:42:42.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75d0ac3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0f5f10 con 0x7f75cc0f51e0 2021-11-20T07:42:42.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:42.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49592/0 (socket says 172.21.15.71:49592) 2021-11-20T07:42:42.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 -- 172.21.15.71:0/204997098 learned_addr learned my addr 172.21.15.71:0/204997098 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:42.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 -- 172.21.15.71:0/204997098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75cc0f51e0 msgr2=0x7f75cc0f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:42.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75cc0f51e0 0x7f75cc0f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:42.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 -- 172.21.15.71:0/204997098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc005040 con 0x7f75cc0f4840 2021-11-20T07:42:42.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.119+0000 7f75cb7fe700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f75bc004e30 tx=0x7f75bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:42.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.120+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75bc018070 con 0x7f75cc0f4840 2021-11-20T07:42:42.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.120+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc0f6190 con 0x7f75cc0f4840 2021-11-20T07:42:42.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.120+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75bc013410 con 0x7f75cc0f4840 2021-11-20T07:42:42.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.120+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75bc01c3c0 con 0x7f75cc0f4840 2021-11-20T07:42:42.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.120+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc0f66f0 con 0x7f75cc0f4840 2021-11-20T07:42:42.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.121+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc045ad0 con 0x7f75cc0f4840 2021-11-20T07:42:42.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.121+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f75bc01d420 con 0x7f75cc0f4840 2021-11-20T07:42:42.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.121+0000 7f75c8ff9700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 0x7f75b805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:42.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.121+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f75bc078060 con 0x7f75cc0f4840 2021-11-20T07:42:42.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.121+0000 7f75c2ffd700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 0x7f75b805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:42.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.122+0000 7f75c2ffd700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 0x7f75b805e7c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f75cc0f5620 tx=0x7f75b4009b10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:42.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.125+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75bc04a820 con 0x7f75cc0f4840 2021-11-20T07:42:42.315 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.314+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75cc027a00 con 0x7f75cc0f4840 2021-11-20T07:42:42.315 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.315+0000 7f75c8ff9700 1 -- 172.21.15.71:0/204997098 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f75bc04de40 con 0x7f75cc0f4840 2021-11-20T07:42:42.316 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:42.316 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:42.318 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 msgr2=0x7f75b805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:42.318 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 0x7f75b805e7c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f75cc0f5620 tx=0x7f75b4009b10).stop 2021-11-20T07:42:42.318 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 msgr2=0x7f75cc0f73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:42.318 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f75bc004e30 tx=0x7f75bc009b30).stop 2021-11-20T07:42:42.319 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 shutdown_connections 2021-11-20T07:42:42.319 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f75b805c300 0x7f75b805e7c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:42.319 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75cc0f4840 0x7f75cc0f73f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:42.319 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 --2- 172.21.15.71:0/204997098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75cc0f51e0 0x7f75cc0f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:42.320 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 >> 172.21.15.71:0/204997098 conn(0x7f75cc0f0360 msgr2=0x7f75cc0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:42.320 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 shutdown_connections 2021-11-20T07:42:42.320 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:42.316+0000 7f75d0ac3700 1 -- 172.21.15.71:0/204997098 wait complete. 2021-11-20T07:42:42.320 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:42.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:42.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:42 smithi112 conmon[36062]: audit 2021-11-20T07:42:42.315083+0000 mon.smithi071 (mon.0) 864 : audit 2021-11-20T07:42:42.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:42 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/204997098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:42.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:42 smithi112 conmon[36062]: cluster 2021-11-20T07:42:42.424808+0000 mgr.smithi071.ynqmwn (mgr.14162) 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:42:42.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:42 smithi071 conmon[33687]: audit 2021-11-20T07:42:42.315083+0000 mon.smithi071 (mon. 2021-11-20T07:42:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:42 smithi071 conmon[33687]: 0) 864 : audit [DBG] from='client.? 172.21.15.71:0/204997098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:42.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:42 smithi071 conmon[33687]: cluster 2021-11-20T07:42:42.424808+0000 mgr.smithi071.ynqmwn (mgr.14162) 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:42:43.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:43.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- 172.21.15.71:0/3499101418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 msgr2=0x7fcd980f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:43.462 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 --2- 172.21.15.71:0/3499101418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd980f6aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fcd940099e0 tx=0x7fcd94004d10).stop 2021-11-20T07:42:43.462 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- 172.21.15.71:0/3499101418 shutdown_connections 2021-11-20T07:42:43.462 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 --2- 172.21.15.71:0/3499101418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd980f7920 0x7fcd980f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 --2- 172.21.15.71:0/3499101418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd980f6aa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- 172.21.15.71:0/3499101418 >> 172.21.15.71:0/3499101418 conn(0x7fcd980f1de0 msgr2=0x7fcd980f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- 172.21.15.71:0/3499101418 shutdown_connections 2021-11-20T07:42:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- 172.21.15.71:0/3499101418 wait complete. 2021-11-20T07:42:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 Processor -- start 2021-11-20T07:42:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 -- start start 2021-11-20T07:42:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.457+0000 7fcda06d0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcda06d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd980f7920 0x7fcd98107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcda06d0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98107b80 con 0x7fcd980f6680 2021-11-20T07:42:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcda06d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98107cc0 con 0x7fcd980f7920 2021-11-20T07:42:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49604/0 (socket says 172.21.15.71:49604) 2021-11-20T07:42:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 -- 172.21.15.71:0/2210467407 learned_addr learned my addr 172.21.15.71:0/2210467407 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 -- 172.21.15.71:0/2210467407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd980f7920 msgr2=0x7fcd98107540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd980f7920 0x7fcd98107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 -- 172.21.15.71:0/2210467407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94009640 con 0x7fcd980f6680 2021-11-20T07:42:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd9e46c700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fcd94007830 tx=0x7fcd94004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9400e030 con 0x7fcd980f6680 2021-11-20T07:42:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd9401d8d0 con 0x7fcd980f6680 2021-11-20T07:42:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9810c700 con 0x7fcd980f6680 2021-11-20T07:42:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd94018070 con 0x7fcd980f6680 2021-11-20T07:42:43.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.458+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9810cbe0 con 0x7fcd980f6680 2021-11-20T07:42:43.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.459+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcd9401da40 con 0x7fcd980f6680 2021-11-20T07:42:43.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.459+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98027a00 con 0x7fcd980f6680 2021-11-20T07:42:43.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.460+0000 7fcd8effd700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 0x7fcd8405e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:43.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.460+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcd940782e0 con 0x7fcd980f6680 2021-11-20T07:42:43.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.460+0000 7fcd9dc6b700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 0x7fcd8405e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.460+0000 7fcd9dc6b700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 0x7fcd8405e750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fcd88009610 tx=0x7fcd88006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.464+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd9404aaa0 con 0x7fcd980f6680 2021-11-20T07:42:43.649 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.648+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd980fee90 con 0x7fcd980f6680 2021-11-20T07:42:43.650 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.649+0000 7fcd8effd700 1 -- 172.21.15.71:0/2210467407 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7fcd9404e0c0 con 0x7fcd980f6680 2021-11-20T07:42:43.650 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:43.651 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:43.652 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 msgr2=0x7fcd8405e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:43.652 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 0x7fcd8405e750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fcd88009610 tx=0x7fcd88006ab0).stop 2021-11-20T07:42:43.652 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 msgr2=0x7fcd98107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:43.652 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fcd94007830 tx=0x7fcd94004d10).stop 2021-11-20T07:42:43.653 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 shutdown_connections 2021-11-20T07:42:43.653 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcd8405c290 0x7fcd8405e750 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:43.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd980f6680 0x7fcd98107050 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:43.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 --2- 172.21.15.71:0/2210467407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd980f7920 0x7fcd98107540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:43.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.650+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 >> 172.21.15.71:0/2210467407 conn(0x7fcd980f1de0 msgr2=0x7fcd980fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:43.654 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.651+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 shutdown_connections 2021-11-20T07:42:43.655 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:43.651+0000 7fcda06d0700 1 -- 172.21.15.71:0/2210467407 wait complete. 2021-11-20T07:42:43.655 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:43.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:43.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:44.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:43 smithi112 conmon[36062]: audit 2021-11-20T07:42:43.649210+0000 mon.smithi071 (mon.0) 865 : 2021-11-20T07:42:44.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:43 smithi112 conmon[36062]: audit [DBG] from='client.? 172.21.15.71:0/2210467407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:44.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:43 smithi071 conmon[33687]: audit 2021-11-20T07:42:43.649210+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:44.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:43 smithi071 conmon[33687]: 865 : audit [DBG] from='client.? 172.21.15.71:0/2210467407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:44.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:44.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 -- 172.21.15.71:0/1757618528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 msgr2=0x7fd0000f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:44.792 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 --2- 172.21.15.71:0/1757618528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0000f7da0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fcfe8004660 tx=0x7fcfe8009b30).stop 2021-11-20T07:42:44.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 -- 172.21.15.71:0/1757618528 shutdown_connections 2021-11-20T07:42:44.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 --2- 172.21.15.71:0/1757618528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0000f7da0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 --2- 172.21.15.71:0/1757618528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd0000f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.793 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 -- 172.21.15.71:0/1757618528 >> 172.21.15.71:0/1757618528 conn(0x7fd0000f1da0 msgr2=0x7fd0000f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 -- 172.21.15.71:0/1757618528 shutdown_connections 2021-11-20T07:42:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 -- 172.21.15.71:0/1757618528 wait complete. 2021-11-20T07:42:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.789+0000 7fd005d89700 1 Processor -- start 2021-11-20T07:42:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fd005d89700 1 -- start start 2021-11-20T07:42:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fd005d89700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd000106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:44.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fd005d89700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:44.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fd005d89700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000107960 con 0x7fd0000f7940 2021-11-20T07:42:44.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fd005d89700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000107aa0 con 0x7fd0000f66a0 2021-11-20T07:42:44.795 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:44.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49616/0 (socket says 172.21.15.71:49616) 2021-11-20T07:42:44.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 -- 172.21.15.71:0/3937929856 learned_addr learned my addr 172.21.15.71:0/3937929856 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:44.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcfff7fe700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd000106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:44.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 -- 172.21.15.71:0/3937929856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 msgr2=0x7fd000106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:44.796 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd000106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.797 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 -- 172.21.15.71:0/3937929856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe8005040 con 0x7fd0000f7940 2021-11-20T07:42:44.797 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcfff7fe700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd000106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:44.797 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcff6ffd700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fcfe8009f50 tx=0x7fcfe8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:44.797 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.790+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe8018070 con 0x7fd0000f7940 2021-11-20T07:42:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.791+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfe8013410 con 0x7fd0000f7940 2021-11-20T07:42:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.791+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00010c4e0 con 0x7fd0000f7940 2021-11-20T07:42:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.791+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe801c3c0 con 0x7fd0000f7940 2021-11-20T07:42:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.791+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00010caa0 con 0x7fd0000f7940 2021-11-20T07:42:44.798 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.792+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000027a00 con 0x7fd0000f7940 2021-11-20T07:42:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.792+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcfe801d420 con 0x7fd0000f7940 2021-11-20T07:42:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.792+0000 7fcffcff9700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 0x7fcfec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.792+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcfe80780c0 con 0x7fd0000f7940 2021-11-20T07:42:44.800 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.792+0000 7fcfff7fe700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 0x7fcfec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.793+0000 7fcfff7fe700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 0x7fcfec05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fd0000f77a0 tx=0x7fcff0009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:44.801 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.796+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe804a880 con 0x7fd0000f7940 2021-11-20T07:42:44.978 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.978+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd00010c670 con 0x7fd0000f7940 2021-11-20T07:42:44.979 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.978+0000 7fcffcff9700 1 -- 172.21.15.71:0/3937929856 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2572 (secure 0 0 0) 0x7fd00010c670 con 0x7fd0000f7940 2021-11-20T07:42:44.980 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:44.980 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:26.421621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":14814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14814":{"gid":14814,"name":"cephfs.smithi071.xtyomb","rank":1,"incarnation":73,"state":"up:active","state_seq":9,"addr":"172.21.15.71:6827/245260831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":245260831},{"type":"v1","addr":"172.21.15.71:6827","nonce":245260831}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 msgr2=0x7fcfec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 0x7fcfec05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fd0000f77a0 tx=0x7fcff0009c00).stop 2021-11-20T07:42:44.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 msgr2=0x7fd0001073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:44.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fcfe8009f50 tx=0x7fcfe8009b30).stop 2021-11-20T07:42:44.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 shutdown_connections 2021-11-20T07:42:44.982 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0000f66a0 0x7fd000106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fcfec05c2a0 0x7fcfec05e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 --2- 172.21.15.71:0/3937929856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0000f7940 0x7fd0001073b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 >> 172.21.15.71:0/3937929856 conn(0x7fd0000f1da0 msgr2=0x7fd0000fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 shutdown_connections 2021-11-20T07:42:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:44.980+0000 7fd005d89700 1 -- 172.21.15.71:0/3937929856 wait complete. 2021-11-20T07:42:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2021-11-20T07:42:44.991 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:44 smithi071 conmon[33687]: cluster 2021-11-20T07:42:44.425261+0000 mgr.smithi071.ynqmwn (mgr.14162) 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:42:44.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:26.421621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 14814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14814': {'gid': 14814, 'name': 'cephfs.smithi071.xtyomb', 'rank': 1, 'incarnation': 73, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.71:6827/245260831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 245260831}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 245260831}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:44.993 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:45.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:44 smithi112 conmon[36062]: cluster 2021-11-20T07:42:44.425261+0000 mgr.smithi071.ynqmwn (mgr.14162) 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:42:45.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:46.022 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: audit 2021-11-20T07:42:44.978550+0000 mon.smithi071 (mon.0) 866 : audit [DBG] from='client.? 172.21.15.71:0/3937929856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:46.023 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: cluster 2021-11-20T07:42: 2021-11-20T07:42:46.023 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: 45.417601+0000 mon.smithi071 (mon.0) 867 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.xtyomb as rank 1 with standby daemon mds.cephfs.smithi112.bxcrgi 2021-11-20T07:42:46.023 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: cluster 2021-11-20T07:42:45.417639+0000 mon.smithi071 (mon.0) 868 : cluster [INF] MDS daemon mds.cephfs.smithi071.xtyomb is removed because it is dead or otherwise unavailable. 2021-11-20T07:42:46.023 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: cluster 2021-11-20T07:42:45.418173+0000 2021-11-20T07:42:46.024 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: mon.smithi071 (mon.0) 869 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:42:46.024 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: cluster 2021-11-20T07:42:46.024 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: 2021-11-20T07:42:45.421697+0000 mon.smithi071 (mon. 2021-11-20T07:42:46.024 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: 0) 870 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T07:42:46.025 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:42:46.025 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: 20T07:42:45.427206+0000 mon.smithi071 (mon 2021-11-20T07:42:46.025 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:45 smithi071 conmon[33687]: .0) 871 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2021-11-20T07:42:46.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: audit 2021-11-20T07:42:44.978550+0000 mon.smithi071 (mon.0) 866 : audit [DBG] from='client.? 172.21.15.71:0/3937929856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:46.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster 2021-11-20T07:42:45.417601+0000 mon.smithi071 (mon.0) 867 : cluster [WRN] Replacing daemon mds.cephfs.smithi071.xtyomb as rank 1 with standby daemon mds.cephfs.smithi112.bxcrgi 2021-11-20T07:42:46.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster 2021-11-20T07:42:45.417639+0000 mon.smithi071 (mon.0) 868 : 2021-11-20T07:42:46.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster [INF] MDS daemon mds.cephfs.smithi071.xtyomb is removed because it is dead or otherwise unavailable. 2021-11-20T07:42:46.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster 2021- 2021-11-20T07:42:46.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: 11-20T07:42:45.418173+0000 mon.smithi071 (mon.0) 869 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:42:46.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster 2021-11-20T07:42:45 2021-11-20T07:42:46.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: .421697+0000 mon.smithi071 (mon.0) 870 : cluster 2021-11-20T07:42:46.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T07:42:46.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster 2021-11-20T07:42 2021-11-20T07:42:46.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: :45.427206+0000 mon.smithi071 (mon.0) 871 : 2021-11-20T07:42:46.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:45 smithi112 conmon[36062]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2021-11-20T07:42:46.124 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 -- 172.21.15.71:0/104916455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f55a0 msgr2=0x7f278c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:46.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 --2- 172.21.15.71:0/104916455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f55a0 0x7f278c0f59c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f277c0099e0 tx=0x7f277c004d10).stop 2021-11-20T07:42:46.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 -- 172.21.15.71:0/104916455 shutdown_connections 2021-11-20T07:42:46.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 --2- 172.21.15.71:0/104916455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f6840 0x7f278c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.125 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 --2- 172.21.15.71:0/104916455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f55a0 0x7f278c0f59c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 -- 172.21.15.71:0/104916455 >> 172.21.15.71:0/104916455 conn(0x7f278c0f0ce0 msgr2=0x7f278c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:46.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 -- 172.21.15.71:0/104916455 shutdown_connections 2021-11-20T07:42:46.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.121+0000 7f27909fb700 1 -- 172.21.15.71:0/104916455 wait complete. 2021-11-20T07:42:46.126 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 Processor -- start 2021-11-20T07:42:46.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 -- start start 2021-11-20T07:42:46.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f55a0 0x7f278c05ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:46.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:46.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c107ba0 con 0x7f278c0f6840 2021-11-20T07:42:46.127 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c10e6b0 con 0x7f278c0f55a0 2021-11-20T07:42:46.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:46.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49626/0 (socket says 172.21.15.71:49626) 2021-11-20T07:42:46.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 -- 172.21.15.71:0/12474944 learned_addr learned my addr 172.21.15.71:0/12474944 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:46.128 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f278a59c700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f55a0 0x7f278c05ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:46.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 -- 172.21.15.71:0/12474944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f55a0 msgr2=0x7f278c05ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:46.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f55a0 0x7f278c05ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 -- 172.21.15.71:0/12474944 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c009640 con 0x7f278c0f6840 2021-11-20T07:42:46.129 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2789d9b700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f27740026d0 tx=0x7f2774009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:46.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27740079c0 con 0x7f278c0f6840 2021-11-20T07:42:46.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f277400f040 con 0x7f278c0f6840 2021-11-20T07:42:46.130 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c10e850 con 0x7f278c0f6840 2021-11-20T07:42:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27740148b0 con 0x7f278c0f6840 2021-11-20T07:42:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.122+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c10ed60 con 0x7f278c0f6840 2021-11-20T07:42:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.123+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c046330 con 0x7f278c0f6840 2021-11-20T07:42:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.124+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2774007b20 con 0x7f278c0f6840 2021-11-20T07:42:46.132 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.124+0000 7f2782ffd700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 0x7f277805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:46.132 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.124+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f2774076290 con 0x7f278c0f6840 2021-11-20T07:42:46.132 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.124+0000 7f278a59c700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 0x7f277805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.124+0000 7f278a59c700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 0x7f277805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f277c00b620 tx=0x7f277c004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.128+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2774048b20 con 0x7f278c0f6840 2021-11-20T07:42:46.312 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.312+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f278c027a00 con 0x7f278c0f6840 2021-11-20T07:42:46.313 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.312+0000 7f2782ffd700 1 -- 172.21.15.71:0/12474944 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f277404c140 con 0x7f278c0f6840 2021-11-20T07:42:46.314 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:46.314 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:45.418056+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:resolve","state_seq":19,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:replay","state_seq":2,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:46.315 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 msgr2=0x7f277805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:46.315 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 0x7f277805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f277c00b620 tx=0x7f277c004d10).stop 2021-11-20T07:42:46.316 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 msgr2=0x7f278c10e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:46.316 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f27740026d0 tx=0x7f2774009de0).stop 2021-11-20T07:42:46.316 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 shutdown_connections 2021-11-20T07:42:46.316 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f278c0f55a0 0x7f278c05ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f277805c2a0 0x7f277805e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 --2- 172.21.15.71:0/12474944 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f278c0f6840 0x7f278c10e1c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 >> 172.21.15.71:0/12474944 conn(0x7f278c0f0ce0 msgr2=0x7f278c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 shutdown_connections 2021-11-20T07:42:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:46.314+0000 7f27909fb700 1 -- 172.21.15.71:0/12474944 wait complete. 2021-11-20T07:42:46.318 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 80 2021-11-20T07:42:46.326 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:45.418056+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:46.327 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:resolve 2021-11-20T07:42:47.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: audit 2021-11-20T07:42:46.312625+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:47.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: 872 : audit [DBG] from='client.? 172.21.15.71:0/12474944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:47.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: cluster 2021-11-20T07:42:46.425785+0000 mgr.smithi071.ynqmwn (mgr.14162) 324 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:42:47.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: cluster 2021-11-20T07:42:46.428839+0000 mon.smithi071 (mon 2021-11-20T07:42:47.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: .0) 873 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:resolve 2021-11-20T07:42:47.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:42:47.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: T07:42:46.428900+0000 mon.smithi071 (mon.0) 874 : 2021-11-20T07:42:47.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:46 smithi112 conmon[36062]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2021-11-20T07:42:47.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: audit 2021-11-20T07:42:46.312625+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:47.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: 872 : audit [DBG] from='client.? 172.21.15.71:0/12474944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:47.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: cluster 2021-11-20T07:42:46.425785+0000 mgr.smithi071.ynqmwn (mgr.14162) 324 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T07:42:47.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: cluster 2021-11-20T07:42:46.428839+0000 mon.smithi071 (mon.0) 873 2021-11-20T07:42:47.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:resolve 2021-11-20T07:42:47.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: cluster 2021-11-20T07:42:46. 2021-11-20T07:42:47.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: 428900+0000 mon.smithi071 (mon.0) 874 : cluster [DBG] 2021-11-20T07:42:47.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:46 smithi071 conmon[33687]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2021-11-20T07:42:47.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:47.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 -- 172.21.15.71:0/203546609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 msgr2=0x7f269c0f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:47.456 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 --2- 172.21.15.71:0/203546609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c0f4e60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f268c004660 tx=0x7f268c009b30).stop 2021-11-20T07:42:47.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 -- 172.21.15.71:0/203546609 shutdown_connections 2021-11-20T07:42:47.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 --2- 172.21.15.71:0/203546609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 0x7f269c0fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 --2- 172.21.15.71:0/203546609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c0f4e60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.457 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 -- 172.21.15.71:0/203546609 >> 172.21.15.71:0/203546609 conn(0x7f269c0f04a0 msgr2=0x7f269c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:47.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 -- 172.21.15.71:0/203546609 shutdown_connections 2021-11-20T07:42:47.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.454+0000 7f26a149a700 1 -- 172.21.15.71:0/203546609 wait complete. 2021-11-20T07:42:47.458 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 Processor -- start 2021-11-20T07:42:47.459 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 -- start start 2021-11-20T07:42:47.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:47.460 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 0x7f269c107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:47.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c107910 con 0x7f269c0f4a40 2021-11-20T07:42:47.461 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f26a149a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c107a50 con 0x7f269c0f53e0 2021-11-20T07:42:47.464 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:47.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49638/0 (socket says 172.21.15.71:49638) 2021-11-20T07:42:47.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 -- 172.21.15.71:0/61799734 learned_addr learned my addr 172.21.15.71:0/61799734 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:47.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f2693fff700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 0x7f269c107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:47.465 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 -- 172.21.15.71:0/61799734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 msgr2=0x7f269c107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:47.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 0x7f269c107360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.455+0000 7f269affd700 1 -- 172.21.15.71:0/61799734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c005040 con 0x7f269c0f4a40 2021-11-20T07:42:47.466 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f269affd700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f268c007a40 tx=0x7f268c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:47.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c018070 con 0x7f269c0f4a40 2021-11-20T07:42:47.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f268c013410 con 0x7f269c0f4a40 2021-11-20T07:42:47.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268c01c620 con 0x7f269c0f4a40 2021-11-20T07:42:47.467 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c10c490 con 0x7f269c0f4a40 2021-11-20T07:42:47.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.456+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c10ca50 con 0x7f269c0f4a40 2021-11-20T07:42:47.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.457+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c027a00 con 0x7f269c0f4a40 2021-11-20T07:42:47.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.457+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f268c01d420 con 0x7f269c0f4a40 2021-11-20T07:42:47.468 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.457+0000 7f26937fe700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 0x7f268805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:47.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.457+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f268c0782a0 con 0x7f269c0f4a40 2021-11-20T07:42:47.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.457+0000 7f2693fff700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 0x7f268805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:47.469 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.458+0000 7f2693fff700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 0x7f268805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f269c0f5860 tx=0x7f2684009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:47.470 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.461+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f268c04a9a0 con 0x7f269c0f4a40 2021-11-20T07:42:47.667 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.666+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f269c0fee60 con 0x7f269c0f4a40 2021-11-20T07:42:47.668 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.666+0000 7f26937fe700 1 -- 172.21.15.71:0/61799734 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f268c04dfc0 con 0x7f269c0f4a40 2021-11-20T07:42:47.669 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:47.669 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:47.426952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:47.671 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 msgr2=0x7f268805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:47.671 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 0x7f268805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f269c0f5860 tx=0x7f2684009040).stop 2021-11-20T07:42:47.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 msgr2=0x7f269c106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:47.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f268c007a40 tx=0x7f268c009b30).stop 2021-11-20T07:42:47.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 shutdown_connections 2021-11-20T07:42:47.672 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f268805c2a0 0x7f268805e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f269c0f4a40 0x7f269c106e70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 --2- 172.21.15.71:0/61799734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f269c0f53e0 0x7f269c107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:47.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 >> 172.21.15.71:0/61799734 conn(0x7f269c0f04a0 msgr2=0x7f269c0f84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:47.673 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 shutdown_connections 2021-11-20T07:42:47.674 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:47.668+0000 7f26a149a700 1 -- 172.21.15.71:0/61799734 wait complete. 2021-11-20T07:42:47.674 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 82 2021-11-20T07:42:47.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:47.426952+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:reconnect', 'state_seq': 25, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:reconnect', 'state_seq': 12, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:47.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:reconnect 2021-11-20T07:42:48.682 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:48.709 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: cluster 2021-11-20T07:42:47.429909+0000 mon.smithi071 (mon.0) 875 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:reconnect 2021-11-20T07:42:48.710 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: cluster 2021-11-20T07:42:47.429954+0000 mon.smithi071 (mon.0) 876 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:reconnect 2021-11-20T07:42:48.710 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: cluster 2021-11-20T07:42:47.430008+0000 mon.smithi071 (mon.0) 877 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2021-11-20T07:42:48.710 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:42:48.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: -11-20T07:42:47 2021-11-20T07:42:48.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: .454455+0000 mds.cephfs.smithi112.iqrmps (mds.0 2021-11-20T07:42:48.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: ) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:42:48.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: cluster 2021- 2021-11-20T07:42:48.711 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: 11-20T07:42:47.454457 2021-11-20T07:42:48.712 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: +0000 mds.cephfs.smithi112.bxcrgi (mds.1) 1 2021-11-20T07:42:48.712 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:42:48.712 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: audit 2021-11-20T07:42:48.712 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: 2021-11-20T07: 2021-11-20T07:42:48.712 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: 42:47.666687+0000 mon.smithi071 (mon.0) 2021-11-20T07:42:48.713 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:48 smithi071 conmon[33687]: 878 : audit [DBG] from='client.? 172.21.15.71:0/61799734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster 2021-11-20T07:42:47.429909+0000 mon.smithi071 (mon.0) 875 : 2021-11-20T07:42:48.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:reconnect 2021-11-20T07:42:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster 2021-11-20T07:42:47.429954+0000 mon.smithi071 (mon.0) 876 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:reconnect 2021-11-20T07:42:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster 2021-11-20T07:42:47.430008+0000 mon.smithi071 (mon 2021-11-20T07:42:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: .0) 877 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:reconnect,1=cephfs.smithi112.bxcrgi=up:reconnect} 2021-11-20T07:42:48.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:42:48.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: -11-20T07:42:47.454455+0000 mds.cephfs.smithi112.iqrmps (mds. 2021-11-20T07:42:48.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: 0) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:42:48.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: cluster 2021-11-20T07:42:47.454457+0000 mds.cephfs.smithi112.bxcrgi ( 2021-11-20T07:42:48.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: mds.1) 1 : cluster [DBG] reconnect by client.24335 v1:172.21.15.71:0/3241013077 after 0.0249993 2021-11-20T07:42:48.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: audit 2021-11-20T07:42:47.666687+0000 2021-11-20T07:42:48.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:48 smithi112 conmon[36062]: mon.smithi071 (mon.0) 878 : audit [DBG] from='client.? 172.21.15.71:0/61799734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:48.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- 172.21.15.71:0/948705698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 msgr2=0x7fce1c0f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:48.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 --2- 172.21.15.71:0/948705698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c0f7da0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fce04004660 tx=0x7fce04009b30).stop 2021-11-20T07:42:48.810 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- 172.21.15.71:0/948705698 shutdown_connections 2021-11-20T07:42:48.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 --2- 172.21.15.71:0/948705698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c0f7da0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:48.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 --2- 172.21.15.71:0/948705698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:48.811 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- 172.21.15.71:0/948705698 >> 172.21.15.71:0/948705698 conn(0x7fce1c0f1da0 msgr2=0x7fce1c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:48.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- 172.21.15.71:0/948705698 shutdown_connections 2021-11-20T07:42:48.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- 172.21.15.71:0/948705698 wait complete. 2021-11-20T07:42:48.812 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 Processor -- start 2021-11-20T07:42:48.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.807+0000 7fce22d28700 1 -- start start 2021-11-20T07:42:48.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:48.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:48.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c1079f0 con 0x7fce1c0f7940 2021-11-20T07:42:48.813 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c107b30 con 0x7fce1c0f66a0 2021-11-20T07:42:48.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:48.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49650/0 (socket says 172.21.15.71:49650) 2021-11-20T07:42:48.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 -- 172.21.15.71:0/154451235 learned_addr learned my addr 172.21.15.71:0/154451235 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:48.814 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce20ac4700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:48.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 -- 172.21.15.71:0/154451235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 msgr2=0x7fce1c106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:48.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:48.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 -- 172.21.15.71:0/154451235 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04005040 con 0x7fce1c0f7940 2021-11-20T07:42:48.815 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce20ac4700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:48.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce1bfff700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fce04007a40 tx=0x7fce04009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:48.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce04018070 con 0x7fce1c0f7940 2021-11-20T07:42:48.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c10c570 con 0x7fce1c0f7940 2021-11-20T07:42:48.816 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce040097d0 con 0x7fce1c0f7940 2021-11-20T07:42:48.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce0401c620 con 0x7fce1c0f7940 2021-11-20T07:42:48.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.808+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c10cad0 con 0x7fce1c0f7940 2021-11-20T07:42:48.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.809+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c027a00 con 0x7fce1c0f7940 2021-11-20T07:42:48.817 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.810+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce0401c890 con 0x7fce1c0f7940 2021-11-20T07:42:48.818 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.810+0000 7fce197fa700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 0x7fce0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:48.819 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.810+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fce040783b0 con 0x7fce1c0f7940 2021-11-20T07:42:48.819 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.810+0000 7fce20ac4700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 0x7fce0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:48.819 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.810+0000 7fce20ac4700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 0x7fce0805e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fce1c0f77a0 tx=0x7fce0c0012e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:48.819 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.814+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce0404ab10 con 0x7fce1c0f7940 2021-11-20T07:42:49.000 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:48.999+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce1c0feed0 con 0x7fce1c0f7940 2021-11-20T07:42:49.001 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.000+0000 7fce197fa700 1 -- 172.21.15.71:0/154451235 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2201 (secure 0 0 0) 0x7fce0404e130 con 0x7fce1c0f7940 2021-11-20T07:42:49.001 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:49.002 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:48.428458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:49.003 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 msgr2=0x7fce0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:49.004 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 0x7fce0805e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fce1c0f77a0 tx=0x7fce0c0012e0).stop 2021-11-20T07:42:49.005 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 msgr2=0x7fce1c1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:49.005 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fce04007a40 tx=0x7fce04009b30).stop 2021-11-20T07:42:49.006 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 shutdown_connections 2021-11-20T07:42:49.006 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce1c0f66a0 0x7fce1c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:49.006 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fce0805c2a0 0x7fce0805e760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:49.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 --2- 172.21.15.71:0/154451235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce1c0f7940 0x7fce1c1073b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:49.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 >> 172.21.15.71:0/154451235 conn(0x7fce1c0f1da0 msgr2=0x7fce1c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:49.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 shutdown_connections 2021-11-20T07:42:49.007 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:49.001+0000 7fce22d28700 1 -- 172.21.15.71:0/154451235 wait complete. 2021-11-20T07:42:49.008 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 83 2021-11-20T07:42:49.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:48.428458+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:rejoin', 'state_seq': 26, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:49.015 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:rejoin 2021-11-20T07:42:49.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: cluster 2021-11-20T07:42:48.426230+0000 mgr.smithi071.ynqmwn (mgr.14162) 325 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:42:49.533 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: cluster 2021-11-20T07:42:48.434519+0000 mon.smithi071 (mon.0 2021-11-20T07:42:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: ) 879 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:rejoin 2021-11-20T07:42:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: cluster 2021-11-20T07:42:48.434585+0000 mon.smithi071 (mon.0) 880 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:rejoin 2021-11-20T07:42:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: cluster 2021-11-20T07:42: 2021-11-20T07:42:49.534 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: 48.434646+0000 mon.smithi071 (mon.0) 881 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2021-11-20T07:42:49.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: cluster 2021-11-20T07:42:49.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: 2021-11-20T07:42: 2021-11-20T07:42:49.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: 48.437474+0000 mon.smithi071 (mon.0 2021-11-20T07:42:49.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: ) 882 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:42:49.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: audit 2021- 2021-11-20T07:42:49.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: 11-20T07:42:49.000195+0000 2021-11-20T07:42:49.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: mon.smithi071 (mon.0) 883 : audit 2021-11-20T07:42:49.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:49 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/154451235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: cluster 2021-11-20T07:42:48.426230+0000 mgr.smithi071.ynqmwn (mgr.14162) 325 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 5.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T07:42:49.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: cluster 2021-11-20T07:42:48.434519+0000 mon.smithi071 (mon.0) 879 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:rejoin 2021-11-20T07:42:49.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: cluster 2021-11-20T07:42:49.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: 2021-11-20T07:42:48. 2021-11-20T07:42:49.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: 434585+0000 mon.smithi071 (mon.0) 880 : cluster [DBG] 2021-11-20T07:42:49.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:rejoin 2021-11-20T07:42:49.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:42:49.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: 42:48.434646+0000 mon.smithi071 (mon.0) 881 2021-11-20T07:42:49.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:rejoin,1=cephfs.smithi112.bxcrgi=up:rejoin} 2021-11-20T07:42:49.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: cluster 2021-11-20T07:42:48 2021-11-20T07:42:49.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: .437474+0000 mon.smithi071 (mon.0) 882 : cluster [INF] 2021-11-20T07:42:49.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: daemon mds.cephfs.smithi112.bxcrgi is now active in filesystem cephfs as rank 1 2021-11-20T07:42:49.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: audit 2021-11-20T07:42:49.000195 2021-11-20T07:42:49.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: +0000 mon.smithi071 (mon.0) 883 : audit [DBG] 2021-11-20T07:42:49.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:49 smithi071 conmon[33687]: from='client.? 172.21.15.71:0/154451235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:50.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:50.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 -- 172.21.15.71:0/130919397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f6680 msgr2=0x7f536c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:50.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 --2- 172.21.15.71:0/130919397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f6680 0x7f536c0f6aa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f5368004660 tx=0x7f5368009b30).stop 2021-11-20T07:42:50.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 -- 172.21.15.71:0/130919397 shutdown_connections 2021-11-20T07:42:50.145 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 --2- 172.21.15.71:0/130919397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f7920 0x7f536c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 --2- 172.21.15.71:0/130919397 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f6680 0x7f536c0f6aa0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 -- 172.21.15.71:0/130919397 >> 172.21.15.71:0/130919397 conn(0x7f536c0f1de0 msgr2=0x7f536c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:50.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 -- 172.21.15.71:0/130919397 shutdown_connections 2021-11-20T07:42:50.146 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.142+0000 7f5373bd3700 1 -- 172.21.15.71:0/130919397 wait complete. 2021-11-20T07:42:50.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 Processor -- start 2021-11-20T07:42:50.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 -- start start 2021-11-20T07:42:50.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f6680 0x7f536c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:50.147 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:50.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c107950 con 0x7f536c0f7920 2021-11-20T07:42:50.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c107a90 con 0x7f536c0f6680 2021-11-20T07:42:50.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:50.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49660/0 (socket says 172.21.15.71:49660) 2021-11-20T07:42:50.148 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 -- 172.21.15.71:0/2901930240 learned_addr learned my addr 172.21.15.71:0/2901930240 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:50.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537196f700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f6680 0x7f536c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:50.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 -- 172.21.15.71:0/2901930240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f6680 msgr2=0x7f536c106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:50.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f6680 0x7f536c106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.149 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 -- 172.21.15.71:0/2901930240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5368005040 con 0x7f536c0f7920 2021-11-20T07:42:50.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f537116e700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f535c00c790 tx=0x7f535c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:50.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f535c00e680 con 0x7f536c0f7920 2021-11-20T07:42:50.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f535c00ec90 con 0x7f536c0f7920 2021-11-20T07:42:50.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c10c530 con 0x7f536c0f7920 2021-11-20T07:42:50.150 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f535c013d60 con 0x7f536c0f7920 2021-11-20T07:42:50.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.143+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c10caf0 con 0x7f536c0f7920 2021-11-20T07:42:50.151 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.144+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c027a00 con 0x7f536c0f7920 2021-11-20T07:42:50.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.145+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f535c01a060 con 0x7f536c0f7920 2021-11-20T07:42:50.152 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.145+0000 7f53627fc700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 0x7f535805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:50.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.145+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f535c076680 con 0x7f536c0f7920 2021-11-20T07:42:50.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.145+0000 7f537196f700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 0x7f535805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:50.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.145+0000 7f537196f700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 0x7f535805e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f5368004e30 tx=0x7f5368009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:50.153 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.149+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f535c048de0 con 0x7f536c0f7920 2021-11-20T07:42:50.341 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.341+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f536c0fee10 con 0x7f536c0f7920 2021-11-20T07:42:50.342 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.341+0000 7f53627fc700 1 -- 172.21.15.71:0/2901930240 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f535c04c400 con 0x7f536c0f7920 2021-11-20T07:42:50.343 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:50.343 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:50.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 msgr2=0x7f535805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:50.344 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 0x7f535805e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f5368004e30 tx=0x7f5368009b30).stop 2021-11-20T07:42:50.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 msgr2=0x7f536c1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:50.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f535c00c790 tx=0x7f535c007d70).stop 2021-11-20T07:42:50.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 shutdown_connections 2021-11-20T07:42:50.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f536c0f6680 0x7f536c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.345 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f535805c2a0 0x7f535805e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 --2- 172.21.15.71:0/2901930240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f536c0f7920 0x7f536c1073a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 >> 172.21.15.71:0/2901930240 conn(0x7f536c0f1de0 msgr2=0x7f536c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 shutdown_connections 2021-11-20T07:42:50.346 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:50.343+0000 7f5373bd3700 1 -- 172.21.15.71:0/2901930240 wait complete. 2021-11-20T07:42:50.347 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:50.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:50.357 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:50.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:50 smithi112 conmon[36062]: cluster 2021-11-20T07:42:49.435820+0000 mon.smithi071 (mon.0) 884 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:clientreplay 2021-11-20T07:42:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:50 smithi112 conmon[36062]: cluster 2021-11-20T07:42:49.435864+0000 mon.smithi071 (mon.0) 885 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:active 2021-11-20T07:42:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:50 smithi112 conmon[36062]: cluster 2021-11-20T07:42:49.435922+0000 mon.smithi071 (mon.0) 886 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:42:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:50 smithi112 conmon[36062]: audit 2021-11-20T07:42:50.341797+0000 mon.smithi071 (mon.0) 887 : audit [DBG] from='client.? 172.21.15.71:0/2901930240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:50.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:50 smithi112 conmon[36062]: cluster 2021-11-20T07:42:50.426607+0000 mgr.smithi071.ynqmwn (mgr.14162) 326 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 5.7 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:42:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: cluster 2021-11-20T07:42:49.435820+0000 mon.smithi071 (mon.0) 884 : cluster [DBG] mds.? [v2:172.21.15.112:6826/845332301,v1:172.21.15.112:6827/845332301] up:clientreplay 2021-11-20T07:42:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: cluster 2021-11-20T07:42:49.435864+0000 mon.smithi071 (mon.0) 885 : cluster [DBG] mds.? [v2:172.21.15.112:6824/1889679177,v1:172.21.15.112:6825/1889679177] up:active 2021-11-20T07:42:50.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: cluster 2021-11-20T07:42 2021-11-20T07:42:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: :49.435922+0000 mon.smithi071 (mon.0) 886 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:clientreplay,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:42:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: audit 2021-11-20T07:42:50. 2021-11-20T07:42:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: 341797+0000 mon.smithi071 (mon.0) 887 : audit 2021-11-20T07:42:50.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/2901930240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: cluster 2021-11-20T07:42:50 2021-11-20T07:42:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: .426607+0000 mgr.smithi071.ynqmwn (mgr.14162) 326 : cluster 2021-11-20T07:42:50.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:50 smithi071 conmon[33687]: [DBG] pgmap v303: 65 pgs: 65 active+clean; 5.7 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-20T07:42:51.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:51.549 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 -- 172.21.15.71:0/2850263755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f6b70 msgr2=0x7f8f880f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:51.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 --2- 172.21.15.71:0/2850263755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f6b70 0x7f8f880f6f90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f8f78004660 tx=0x7f8f78009b30).stop 2021-11-20T07:42:51.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 -- 172.21.15.71:0/2850263755 shutdown_connections 2021-11-20T07:42:51.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 --2- 172.21.15.71:0/2850263755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f5870 0x7f8f880f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:51.550 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 --2- 172.21.15.71:0/2850263755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f6b70 0x7f8f880f6f90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:51.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.545+0000 7f8f8e906700 1 -- 172.21.15.71:0/2850263755 >> 172.21.15.71:0/2850263755 conn(0x7f8f880f0440 msgr2=0x7f8f880f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:51.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 -- 172.21.15.71:0/2850263755 shutdown_connections 2021-11-20T07:42:51.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 -- 172.21.15.71:0/2850263755 wait complete. 2021-11-20T07:42:51.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 Processor -- start 2021-11-20T07:42:51.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 -- start start 2021-11-20T07:42:51.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:51.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f6b70 0x7f8f881073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f88107a10 con 0x7f8f880f5870 2021-11-20T07:42:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f8e906700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f88107b50 con 0x7f8f880f6b70 2021-11-20T07:42:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f87fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f87fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49674/0 (socket says 172.21.15.71:49674) 2021-11-20T07:42:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.546+0000 7f8f87fff700 1 -- 172.21.15.71:0/867515353 learned_addr learned my addr 172.21.15.71:0/867515353 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f877fe700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f6b70 0x7f8f881073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f87fff700 1 -- 172.21.15.71:0/867515353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f6b70 msgr2=0x7f8f881073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f87fff700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f6b70 0x7f8f881073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f87fff700 1 -- 172.21.15.71:0/867515353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78005040 con 0x7f8f880f5870 2021-11-20T07:42:51.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f87fff700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f8f78009f50 tx=0x7f8f78009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:51.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f78018070 con 0x7f8f880f5870 2021-11-20T07:42:51.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f78013410 con 0x7f8f880f5870 2021-11-20T07:42:51.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8810c590 con 0x7f8f880f5870 2021-11-20T07:42:51.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f7801c620 con 0x7f8f880f5870 2021-11-20T07:42:51.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.547+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8810caf0 con 0x7f8f880f5870 2021-11-20T07:42:51.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.548+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f88027a00 con 0x7f8f880f5870 2021-11-20T07:42:51.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.548+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f7801d420 con 0x7f8f880f5870 2021-11-20T07:42:51.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.549+0000 7f8f84ff9700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 0x7f8f7005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:51.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.549+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f8f780782d0 con 0x7f8f880f5870 2021-11-20T07:42:51.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.549+0000 7f8f877fe700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 0x7f8f7005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:51.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.549+0000 7f8f877fe700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 0x7f8f7005e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f8f880f6940 tx=0x7f8f7c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:51.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.553+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f7804a9d0 con 0x7f8f880f5870 2021-11-20T07:42:51.739 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.737+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f880fef00 con 0x7f8f880f5870 2021-11-20T07:42:51.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.738+0000 7f8f84ff9700 1 -- 172.21.15.71:0/867515353 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f8f7804dff0 con 0x7f8f880f5870 2021-11-20T07:42:51.741 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:51.741 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:51.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 msgr2=0x7f8f7005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:51.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 0x7f8f7005e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f8f880f6940 tx=0x7f8f7c008040).stop 2021-11-20T07:42:51.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 msgr2=0x7f8f88106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:51.744 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f8f78009f50 tx=0x7f8f78009b30).stop 2021-11-20T07:42:51.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 shutdown_connections 2021-11-20T07:42:51.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8f7005c2a0 0x7f8f7005e760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:51.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f880f5870 0x7f8f88106ee0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:51.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 --2- 172.21.15.71:0/867515353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f880f6b70 0x7f8f881073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:51.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 >> 172.21.15.71:0/867515353 conn(0x7f8f880f0440 msgr2=0x7f8f880fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:51.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 shutdown_connections 2021-11-20T07:42:51.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:51.739+0000 7f8f8e906700 1 -- 172.21.15.71:0/867515353 wait complete. 2021-11-20T07:42:51.750 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:51.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:51.754 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:52 smithi071 conmon[33687]: audit 2021-11-20T07:42:51.739174+0000 mon.smithi071 (mon.0) 888 : 2021-11-20T07:42:52.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:52 smithi071 conmon[33687]: audit [DBG] from='client.? 172.21.15.71:0/867515353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:52.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:52.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:52 smithi112 conmon[36062]: audit 2021-11-20T07:42:51.739174+0000 mon.smithi071 (mon.0) 888 : audit [DBG] from='client.? 172.21.15.71:0/867515353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:52.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 -- 172.21.15.71:0/2993188970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 msgr2=0x7fca000f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:52.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 --2- 172.21.15.71:0/2993188970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca000f6f90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fc9f0004660 tx=0x7fc9f0009b30).stop 2021-11-20T07:42:52.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 -- 172.21.15.71:0/2993188970 shutdown_connections 2021-11-20T07:42:52.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 --2- 172.21.15.71:0/2993188970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:52.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 --2- 172.21.15.71:0/2993188970 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca000f6f90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:52.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 -- 172.21.15.71:0/2993188970 >> 172.21.15.71:0/2993188970 conn(0x7fca000f0440 msgr2=0x7fca000f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:52.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 -- 172.21.15.71:0/2993188970 shutdown_connections 2021-11-20T07:42:52.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.877+0000 7fca05ece700 1 -- 172.21.15.71:0/2993188970 wait complete. 2021-11-20T07:42:52.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 Processor -- start 2021-11-20T07:42:52.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 -- start start 2021-11-20T07:42:52.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:52.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:52.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0010c1a0 con 0x7fca000f6b70 2021-11-20T07:42:52.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fca05ece700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0010c2e0 con 0x7fca000f5870 2021-11-20T07:42:52.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:52.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49688/0 (socket says 172.21.15.71:49688) 2021-11-20T07:42:52.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 -- 172.21.15.71:0/3245742057 learned_addr learned my addr 172.21.15.71:0/3245742057 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:52.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9ff7fe700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:52.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 -- 172.21.15.71:0/3245742057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 msgr2=0x7fca000f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:52.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:52.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 -- 172.21.15.71:0/3245742057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f0005040 con 0x7fca000f6b70 2021-11-20T07:42:52.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9ff7fe700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:52.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.878+0000 7fc9feffd700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fc9f400c7c0 tx=0x7fc9f4007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:52.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.879+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9f400e680 con 0x7fca000f6b70 2021-11-20T07:42:52.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.879+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0010c540 con 0x7fca000f6b70 2021-11-20T07:42:52.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.879+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9f400ec90 con 0x7fca000f6b70 2021-11-20T07:42:52.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.879+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9f4012ce0 con 0x7fca000f6b70 2021-11-20T07:42:52.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.879+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0010cb00 con 0x7fca000f6b70 2021-11-20T07:42:52.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.880+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca00027a00 con 0x7fca000f6b70 2021-11-20T07:42:52.888 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.880+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc9f4010070 con 0x7fca000f6b70 2021-11-20T07:42:52.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.880+0000 7fc9e7fff700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 0x7fc9e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:52.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.880+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fc9f40764f0 con 0x7fca000f6b70 2021-11-20T07:42:52.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.880+0000 7fc9ff7fe700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 0x7fc9e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:52.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.881+0000 7fc9ff7fe700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 0x7fc9e805e760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fc9f0004e30 tx=0x7fc9f0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:52.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:52.884+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9f4048c50 con 0x7fca000f6b70 2021-11-20T07:42:53.070 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.068+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca000fdd60 con 0x7fca000f6b70 2021-11-20T07:42:53.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.069+0000 7fc9e7fff700 1 -- 172.21.15.71:0/3245742057 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7fc9f404c270 con 0x7fca000f6b70 2021-11-20T07:42:53.071 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:53.072 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:53.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 msgr2=0x7fc9e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:53.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 0x7fc9e805e760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fc9f0004e30 tx=0x7fc9f0009b30).stop 2021-11-20T07:42:53.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 msgr2=0x7fca0010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:53.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fc9f400c7c0 tx=0x7fc9f4007d90).stop 2021-11-20T07:42:53.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 shutdown_connections 2021-11-20T07:42:53.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca000f5870 0x7fca000f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:53.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc9e805c2a0 0x7fc9e805e760 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:53.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 --2- 172.21.15.71:0/3245742057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca000f6b70 0x7fca0010bcb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:53.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 >> 172.21.15.71:0/3245742057 conn(0x7fca000f0440 msgr2=0x7fca000fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:53.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 shutdown_connections 2021-11-20T07:42:53.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:53.070+0000 7fca05ece700 1 -- 172.21.15.71:0/3245742057 wait complete. 2021-11-20T07:42:53.076 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:53.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:53.084 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:53.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:53 smithi112 conmon[36062]: cluster 2021-11-20T07:42:52.427001+0000 mgr.smithi071.ynqmwn (mgr.14162) 327 2021-11-20T07:42:53.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:53 smithi112 conmon[36062]: : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:53.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:53 smithi112 conmon[36062]: audit 2021-11-20T07:42:53.070137+0000 mon.smithi071 (mon.0) 889 : audit [DBG] from='client.? 172.21.15.71:0/3245742057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:53.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:53 smithi071 conmon[33687]: cluster 2021-11-20T07:42:52.427001+0000 mgr.smithi071.ynqmwn (mgr.14162) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:53.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:53 smithi071 conmon[33687]: audit 2021-11-20T07:42:53.070137+0000 mon.smithi071 (mon.0) 889 : audit [DBG] from='client.? 172.21.15.71:0/3245742057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:54.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:54.215 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 -- 172.21.15.71:0/523453788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 msgr2=0x7f6b440f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:54.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 --2- 172.21.15.71:0/523453788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f6fd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f6b2c004660 tx=0x7f6b2c009b30).stop 2021-11-20T07:42:54.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 -- 172.21.15.71:0/523453788 shutdown_connections 2021-11-20T07:42:54.216 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 --2- 172.21.15.71:0/523453788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 0x7f6b440f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 --2- 172.21.15.71:0/523453788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f6fd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 -- 172.21.15.71:0/523453788 >> 172.21.15.71:0/523453788 conn(0x7f6b440f0440 msgr2=0x7f6b440f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:54.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 -- 172.21.15.71:0/523453788 shutdown_connections 2021-11-20T07:42:54.217 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.210+0000 7f6b4b137700 1 -- 172.21.15.71:0/523453788 wait complete. 2021-11-20T07:42:54.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 Processor -- start 2021-11-20T07:42:54.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 -- start start 2021-11-20T07:42:54.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:54.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 0x7f6b4410bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:54.218 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4410c200 con 0x7f6b440f6bb0 2021-11-20T07:42:54.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b4b137700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4410c340 con 0x7f6b440f7550 2021-11-20T07:42:54.219 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:54.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49696/0 (socket says 172.21.15.71:49696) 2021-11-20T07:42:54.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 -- 172.21.15.71:0/1245320180 learned_addr learned my addr 172.21.15.71:0/1245320180 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:54.220 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b43fff700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 0x7f6b4410bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 -- 172.21.15.71:0/1245320180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 msgr2=0x7f6b4410bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 0x7f6b4410bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.211+0000 7f6b48ed3700 1 -- 172.21.15.71:0/1245320180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c005040 con 0x7f6b440f6bb0 2021-11-20T07:42:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b48ed3700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f6b2c007a60 tx=0x7f6b2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:54.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b2c018070 con 0x7f6b440f6bb0 2021-11-20T07:42:54.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b2c013410 con 0x7f6b440f6bb0 2021-11-20T07:42:54.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4410c540 con 0x7f6b440f6bb0 2021-11-20T07:42:54.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b2c01c620 con 0x7f6b440f6bb0 2021-11-20T07:42:54.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.212+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4410ca70 con 0x7f6b440f6bb0 2021-11-20T07:42:54.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.213+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b44046330 con 0x7f6b440f6bb0 2021-11-20T07:42:54.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.213+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6b2c01c890 con 0x7f6b440f6bb0 2021-11-20T07:42:54.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.213+0000 7f6b417fa700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 0x7f6b3005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:54.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.213+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f6b2c0782c0 con 0x7f6b440f6bb0 2021-11-20T07:42:54.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.213+0000 7f6b43fff700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 0x7f6b3005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:54.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.214+0000 7f6b43fff700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 0x7f6b3005e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6b340096c0 tx=0x7f6b34009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:54.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.217+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b2c04aa20 con 0x7f6b440f6bb0 2021-11-20T07:42:54.407 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.406+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b44027a00 con 0x7f6b440f6bb0 2021-11-20T07:42:54.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.406+0000 7f6b417fa700 1 -- 172.21.15.71:0/1245320180 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f6b2c04e040 con 0x7f6b440f6bb0 2021-11-20T07:42:54.409 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:54.409 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:54.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 msgr2=0x7f6b3005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:54.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 0x7f6b3005e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6b340096c0 tx=0x7f6b34009be0).stop 2021-11-20T07:42:54.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 msgr2=0x7f6b440f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:54.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f6b2c007a60 tx=0x7f6b2c009b30).stop 2021-11-20T07:42:54.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 shutdown_connections 2021-11-20T07:42:54.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6b3005c2a0 0x7f6b3005e760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b440f6bb0 0x7f6b440f52f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 --2- 172.21.15.71:0/1245320180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6b440f7550 0x7f6b4410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:54.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 >> 172.21.15.71:0/1245320180 conn(0x7f6b440f0440 msgr2=0x7f6b440fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:54.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 shutdown_connections 2021-11-20T07:42:54.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:54.408+0000 7f6b4b137700 1 -- 172.21.15.71:0/1245320180 wait complete. 2021-11-20T07:42:54.414 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:54.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:54.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:54.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:54 smithi112 conmon[36062]: audit 2021-11-20T07:42:54.407480+0000 mon.smithi071 (mon.0) 890 2021-11-20T07:42:54.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:54 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/1245320180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:54.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:54 smithi071 conmon[33687]: audit 2021-11-20T07:42:54.407480+0000 mon.smithi071 (mon.0) 890 : audit [DBG] from='client.? 172.21.15.71:0/1245320180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:55.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:55.551 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- 172.21.15.71:0/4094144491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 msgr2=0x7f85640f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:55.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 --2- 172.21.15.71:0/4094144491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f85640f6ca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f854c004660 tx=0x7f854c009b30).stop 2021-11-20T07:42:55.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- 172.21.15.71:0/4094144491 shutdown_connections 2021-11-20T07:42:55.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 --2- 172.21.15.71:0/4094144491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f85640f6ca0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 --2- 172.21.15.71:0/4094144491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 0x7f85640f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- 172.21.15.71:0/4094144491 >> 172.21.15.71:0/4094144491 conn(0x7f85640f0ce0 msgr2=0x7f85640f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:55.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- 172.21.15.71:0/4094144491 shutdown_connections 2021-11-20T07:42:55.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- 172.21.15.71:0/4094144491 wait complete. 2021-11-20T07:42:55.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 Processor -- start 2021-11-20T07:42:55.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.546+0000 7f85695ae700 1 -- start start 2021-11-20T07:42:55.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 0x7f856405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:55.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:55.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856410c180 con 0x7f85640f6840 2021-11-20T07:42:55.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856410c2c0 con 0x7f85640f55a0 2021-11-20T07:42:55.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:55.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49710/0 (socket says 172.21.15.71:49710) 2021-11-20T07:42:55.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 -- 172.21.15.71:0/2525187581 learned_addr learned my addr 172.21.15.71:0/2525187581 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:55.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 -- 172.21.15.71:0/2525187581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 msgr2=0x7f856405ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:55.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f8562ffd700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 0x7f856405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:55.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 0x7f856405ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 -- 172.21.15.71:0/2525187581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c005040 con 0x7f85640f6840 2021-11-20T07:42:55.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85627fc700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f854c007a60 tx=0x7f854c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:55.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854c018070 con 0x7f85640f6840 2021-11-20T07:42:55.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856410c4c0 con 0x7f85640f6840 2021-11-20T07:42:55.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f854c013410 con 0x7f85640f6840 2021-11-20T07:42:55.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854c01c620 con 0x7f85640f6840 2021-11-20T07:42:55.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.547+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856410ca20 con 0x7f85640f6840 2021-11-20T07:42:55.559 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.548+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8564027a00 con 0x7f85640f6840 2021-11-20T07:42:55.559 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.549+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f854c01c890 con 0x7f85640f6840 2021-11-20T07:42:55.559 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.549+0000 7f855b7fe700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 0x7f855005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:55.559 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.549+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f854c0782d0 con 0x7f85640f6840 2021-11-20T07:42:55.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.549+0000 7f8562ffd700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 0x7f855005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:55.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.549+0000 7f8562ffd700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 0x7f855005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f85640f66a0 tx=0x7f8554009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:55.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.553+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f854c04aa80 con 0x7f85640f6840 2021-11-20T07:42:55.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.736+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85640fdd30 con 0x7f85640f6840 2021-11-20T07:42:55.738 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.737+0000 7f855b7fe700 1 -- 172.21.15.71:0/2525187581 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f854c04e0a0 con 0x7f85640f6840 2021-11-20T07:42:55.739 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:55.739 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:55.740 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 msgr2=0x7f855005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:55.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 0x7f855005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f85640f66a0 tx=0x7f8554009040).stop 2021-11-20T07:42:55.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 msgr2=0x7f856410bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:55.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f854c007a60 tx=0x7f854c009b30).stop 2021-11-20T07:42:55.741 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 shutdown_connections 2021-11-20T07:42:55.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85640f55a0 0x7f856405ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f855005c2a0 0x7f855005e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 --2- 172.21.15.71:0/2525187581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f85640f6840 0x7f856410bc90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:55.742 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 >> 172.21.15.71:0/2525187581 conn(0x7f85640f0ce0 msgr2=0x7f85640fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:55.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 shutdown_connections 2021-11-20T07:42:55.743 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:55.738+0000 7f85695ae700 1 -- 172.21.15.71:0/2525187581 wait complete. 2021-11-20T07:42:55.743 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:55.750 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:55 smithi071 conmon[33687]: cluster 2021-11-20T07:42:54.427492+0000 mgr.smithi071.ynqmwn (mgr.14162) 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:55.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:55.752 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:55 smithi112 conmon[36062]: cluster 2021-11-20T07:42:54.427492+0000 mgr.smithi071.ynqmwn ( 2021-11-20T07:42:55.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:55 smithi112 conmon[36062]: mgr.14162) 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-20T07:42:56.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:56.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:56 smithi112 conmon[36062]: audit 2021-11-20T07:42:55.738047+0000 mon.smithi071 (mon.0) 891 : audit [DBG] from='client.? 172.21.15.71:0/2525187581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:56.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:56 smithi112 conmon[36062]: cluster 2021-11-20T07:42:56.428057+0000 mgr.smithi071.ynqmwn (mgr.14162) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 372 B/s wr, 1 op/s 2021-11-20T07:42:56.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:56 smithi071 conmon[33687]: audit 2021-11-20T07:42:55.738047+0000 mon.smithi071 (mon.0) 891 : audit [DBG] from='client.? 172.21.15.71:0/2525187581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:56.786 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:56 smithi071 conmon[33687]: cluster 2021-11-20T07:42:56.428057+0000 mgr.smithi071.ynqmwn (mgr.14162) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 372 B/s wr, 1 op/s 2021-11-20T07:42:56.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.882+0000 7f6d5b59e700 1 -- 172.21.15.71:0/2784161164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f6ab0 msgr2=0x7f6d5c0f6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:56.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.882+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/2784161164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f6ed0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f6d4c004660 tx=0x7f6d4c009b30).stop 2021-11-20T07:42:56.893 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- 172.21.15.71:0/2784161164 shutdown_connections 2021-11-20T07:42:56.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/2784161164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:56.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/2784161164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f6ed0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:56.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- 172.21.15.71:0/2784161164 >> 172.21.15.71:0/2784161164 conn(0x7f6d5c0f0360 msgr2=0x7f6d5c0f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:56.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- 172.21.15.71:0/2784161164 shutdown_connections 2021-11-20T07:42:56.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- 172.21.15.71:0/2784161164 wait complete. 2021-11-20T07:42:56.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 Processor -- start 2021-11-20T07:42:56.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- start start 2021-11-20T07:42:56.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:56.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:56.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0f8100 con 0x7f6d5c0f7450 2021-11-20T07:42:56.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5b59e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0f8240 con 0x7f6d5c0f6ab0 2021-11-20T07:42:56.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d59d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:56.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d59d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49720/0 (socket says 172.21.15.71:49720) 2021-11-20T07:42:56.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d59d9b700 1 -- 172.21.15.71:0/3889233605 learned_addr learned my addr 172.21.15.71:0/3889233605 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:56.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.883+0000 7f6d5a59c700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:56.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d59d9b700 1 -- 172.21.15.71:0/3889233605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f6ab0 msgr2=0x7f6d5c0f9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:56.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d59d9b700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:56.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d59d9b700 1 -- 172.21.15.71:0/3889233605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c005040 con 0x7f6d5c0f7450 2021-11-20T07:42:56.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d59d9b700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f6d500026d0 tx=0x7f6d50009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:56.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d500079c0 con 0x7f6d5c0f7450 2021-11-20T07:42:56.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d5000f040 con 0x7f6d5c0f7450 2021-11-20T07:42:56.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c0f8520 con 0x7f6d5c0f7450 2021-11-20T07:42:56.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d500148b0 con 0x7f6d5c0f7450 2021-11-20T07:42:56.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.884+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c10edb0 con 0x7f6d5c0f7450 2021-11-20T07:42:56.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.885+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c045ad0 con 0x7f6d5c0f7450 2021-11-20T07:42:56.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.885+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6d50007b20 con 0x7f6d5c0f7450 2021-11-20T07:42:56.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.885+0000 7f6d4affd700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 0x7f6d4405e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:56.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.886+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f6d50076460 con 0x7f6d5c0f7450 2021-11-20T07:42:56.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.886+0000 7f6d5a59c700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 0x7f6d4405e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:56.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.886+0000 7f6d5a59c700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 0x7f6d4405e810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6d5c0f94b0 tx=0x7f6d4c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:56.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:56.889+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d50048b00 con 0x7f6d5c0f7450 2021-11-20T07:42:57.075 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.073+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d5c0fdc40 con 0x7f6d5c0f7450 2021-11-20T07:42:57.076 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.074+0000 7f6d4affd700 1 -- 172.21.15.71:0/3889233605 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f6d5004c120 con 0x7f6d5c0f7450 2021-11-20T07:42:57.076 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:57.077 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:57.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.075+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 msgr2=0x7f6d4405e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:57.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.075+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 0x7f6d4405e810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6d5c0f94b0 tx=0x7f6d4c009b30).stop 2021-11-20T07:42:57.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 msgr2=0x7f6d5c0f7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:57.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f6d500026d0 tx=0x7f6d50009de0).stop 2021-11-20T07:42:57.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 shutdown_connections 2021-11-20T07:42:57.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d5c0f6ab0 0x7f6d5c0f9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:57.079 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f6d4405c350 0x7f6d4405e810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 --2- 172.21.15.71:0/3889233605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d5c0f7450 0x7f6d5c0f7ba0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 >> 172.21.15.71:0/3889233605 conn(0x7f6d5c0f0360 msgr2=0x7f6d5c0f4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 shutdown_connections 2021-11-20T07:42:57.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:57.076+0000 7f6d5b59e700 1 -- 172.21.15.71:0/3889233605 wait complete. 2021-11-20T07:42:57.081 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:57.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:57.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:57.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:57 smithi112 conmon[36062]: audit 2021-11-20T07:42:57.075391+0000 mon.smithi071 (mon.0) 892 : audit [DBG] from='client.? 172.21.15.71:0/3889233605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:57.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:57 smithi071 conmon[33687]: audit 2021-11-20T07:42:57.075391+0000 mon.smithi071 (mon.0) 892 2021-11-20T07:42:57.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:57 smithi071 conmon[33687]: : audit [DBG] from='client.? 172.21.15.71:0/3889233605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:58.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:58.241 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 -- 172.21.15.71:0/2999615978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f7920 msgr2=0x7fd0ac0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:58.241 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/2999615978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac0f7d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fd0a0004660 tx=0x7fd0a0009b30).stop 2021-11-20T07:42:58.241 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 -- 172.21.15.71:0/2999615978 shutdown_connections 2021-11-20T07:42:58.241 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/2999615978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac0f7d80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/2999615978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:42:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 -- 172.21.15.71:0/2999615978 >> 172.21.15.71:0/2999615978 conn(0x7fd0ac0f1de0 msgr2=0x7fd0ac0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 -- 172.21.15.71:0/2999615978 shutdown_connections 2021-11-20T07:42:58.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.234+0000 7fd0b16a4700 1 -- 172.21.15.71:0/2999615978 wait complete. 2021-11-20T07:42:58.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 Processor -- start 2021-11-20T07:42:58.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 -- start start 2021-11-20T07:42:58.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:58.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:58.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1079e0 con 0x7fd0ac0f6680 2021-11-20T07:42:58.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0b16a4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac107b20 con 0x7fd0ac0f7920 2021-11-20T07:42:58.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0aaffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:58.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0aaffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49730/0 (socket says 172.21.15.71:49730) 2021-11-20T07:42:58.244 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0aaffd700 1 -- 172.21.15.71:0/4165227480 learned_addr learned my addr 172.21.15.71:0/4165227480 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:58.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.235+0000 7fd0aa7fc700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:58.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0aaffd700 1 -- 172.21.15.71:0/4165227480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 msgr2=0x7fd0ac1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:58.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0aaffd700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:58.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0aaffd700 1 -- 172.21.15.71:0/4165227480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a0005040 con 0x7fd0ac0f6680 2021-11-20T07:42:58.245 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0aa7fc700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T07:42:58.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0aaffd700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fd09c0093f0 tx=0x7fd09c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:58.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd09c013640 con 0x7fd0ac0f6680 2021-11-20T07:42:58.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac10c5c0 con 0x7fd0ac0f6680 2021-11-20T07:42:58.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd09c009e40 con 0x7fd0ac0f6680 2021-11-20T07:42:58.246 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd09c00dbf0 con 0x7fd0ac0f6680 2021-11-20T07:42:58.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.236+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac10cb80 con 0x7fd0ac0f6680 2021-11-20T07:42:58.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.237+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ac027a00 con 0x7fd0ac0f6680 2021-11-20T07:42:58.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.237+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd09c01a060 con 0x7fd0ac0f6680 2021-11-20T07:42:58.247 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.237+0000 7fd0937fe700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 0x7fd09405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:58.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.238+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fd09c076490 con 0x7fd0ac0f6680 2021-11-20T07:42:58.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.238+0000 7fd0aa7fc700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 0x7fd09405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:58.248 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.238+0000 7fd0aa7fc700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 0x7fd09405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fd0a0013580 tx=0x7fd0a0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:58.249 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.241+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd09c048b90 con 0x7fd0ac0f6680 2021-11-20T07:42:58.430 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.428+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0ac0feeb0 con 0x7fd0ac0f6680 2021-11-20T07:42:58.430 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.428+0000 7fd0937fe700 1 -- 172.21.15.71:0/4165227480 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7fd09c04c1b0 con 0x7fd0ac0f6680 2021-11-20T07:42:58.431 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:58.431 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:58.432 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 msgr2=0x7fd09405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:58.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 0x7fd09405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fd0a0013580 tx=0x7fd0a0009b30).stop 2021-11-20T07:42:58.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 msgr2=0x7fd0ac106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:58.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fd09c0093f0 tx=0x7fd09c006ec0).stop 2021-11-20T07:42:58.433 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 shutdown_connections 2021-11-20T07:42:58.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fd09405c2a0 0x7fd09405e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:58.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0ac0f6680 0x7fd0ac106eb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:58.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 --2- 172.21.15.71:0/4165227480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ac0f7920 0x7fd0ac1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:58.434 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 >> 172.21.15.71:0/4165227480 conn(0x7fd0ac0f1de0 msgr2=0x7fd0ac0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:58.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 shutdown_connections 2021-11-20T07:42:58.435 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:58.430+0000 7fd0b16a4700 1 -- 172.21.15.71:0/4165227480 wait complete. 2021-11-20T07:42:58.435 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:58.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:58.444 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:42:58.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:58 smithi112 conmon[36062]: cluster 2021-11-20T07:42:58.428485+0000 mgr.smithi071.ynqmwn (mgr.14162) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:58 smithi112 conmon[36062]: audit 2021-11-20T 2021-11-20T07:42:58.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:58 smithi112 conmon[36062]: 07:42:58.429468+0000 mon.smithi071 (mon.0) 893 : audit [DBG] from='client.? 172.21.15.71:0/4165227480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:58.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:58 smithi071 conmon[33687]: cluster 2021-11-20T07:42:58.428485+0000 mgr.smithi071.ynqmwn (mgr. 2021-11-20T07:42:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:58 smithi071 conmon[33687]: 14162) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-20T07:42:58.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:58 smithi071 conmon[33687]: audit 2021-11-20T07:42:58.429468+0000 mon.smithi071 (mon.0) 893 : audit [DBG] from='client.? 172.21.15.71:0/4165227480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:42:59.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:42:59.575 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.570+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2369055706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 msgr2=0x7f8d4c0f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:59.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.570+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2369055706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f4c40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f8d3c004660 tx=0x7f8d3c009b30).stop 2021-11-20T07:42:59.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.570+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2369055706 shutdown_connections 2021-11-20T07:42:59.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.570+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2369055706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 0x7f8d4c0f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.570+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2369055706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f4c40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2369055706 >> 172.21.15.71:0/2369055706 conn(0x7f8d4c0f0320 msgr2=0x7f8d4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2369055706 shutdown_connections 2021-11-20T07:42:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2369055706 wait complete. 2021-11-20T07:42:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 Processor -- start 2021-11-20T07:42:59.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- start start 2021-11-20T07:42:59.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:59.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 0x7f8d4c0f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:59.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0f5db0 con 0x7f8d4c0f4820 2021-11-20T07:42:59.580 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.571+0000 7f8d4bfff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0f5ef0 con 0x7f8d4c0f51c0 2021-11-20T07:42:59.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.572+0000 7f8d4affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:59.581 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.572+0000 7f8d4affd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49740/0 (socket says 172.21.15.71:49740) 2021-11-20T07:42:59.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.572+0000 7f8d4affd700 1 -- 172.21.15.71:0/2796914581 learned_addr learned my addr 172.21.15.71:0/2796914581 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:42:59.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.572+0000 7f8d4a7fc700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 0x7f8d4c0f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:59.582 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4affd700 1 -- 172.21.15.71:0/2796914581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 msgr2=0x7f8d4c0f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4affd700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 0x7f8d4c0f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4affd700 1 -- 172.21.15.71:0/2796914581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c005040 con 0x7f8d4c0f4820 2021-11-20T07:42:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4affd700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8d3c009f50 tx=0x7f8d3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:59.583 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d3c018070 con 0x7f8d4c0f4820 2021-11-20T07:42:59.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c0f6170 con 0x7f8d4c0f4820 2021-11-20T07:42:59.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d3c013410 con 0x7f8d4c0f4820 2021-11-20T07:42:59.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d3c01c620 con 0x7f8d4c0f4820 2021-11-20T07:42:59.584 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.573+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c0f66d0 con 0x7f8d4c0f4820 2021-11-20T07:42:59.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.574+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c045ad0 con 0x7f8d4c0f4820 2021-11-20T07:42:59.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.575+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8d3c01d420 con 0x7f8d4c0f4820 2021-11-20T07:42:59.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.575+0000 7f8d337fe700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 0x7f8d3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:42:59.586 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.575+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f8d3c0782e0 con 0x7f8d4c0f4820 2021-11-20T07:42:59.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.575+0000 7f8d4a7fc700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 0x7f8d3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:42:59.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.575+0000 7f8d4a7fc700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 0x7f8d3405e7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f8d400041d0 tx=0x7f8d40009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:42:59.587 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.579+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d3c04aa40 con 0x7f8d4c0f4820 2021-11-20T07:42:59.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.761+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d4c027a00 con 0x7f8d4c0f4820 2021-11-20T07:42:59.763 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.761+0000 7f8d337fe700 1 -- 172.21.15.71:0/2796914581 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2207 (secure 0 0 0) 0x7f8d3c04e060 con 0x7f8d4c0f4820 2021-11-20T07:42:59.764 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:42:59.764 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:42:49.432478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24451,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":78,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.112:6827/845332301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":845332301},{"type":"v1","addr":"172.21.15.112:6827","nonce":845332301}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:42:59.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 msgr2=0x7f8d3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:59.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 0x7f8d3405e7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f8d400041d0 tx=0x7f8d40009040).stop 2021-11-20T07:42:59.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 msgr2=0x7f8d4c0f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:42:59.766 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8d3c009f50 tx=0x7f8d3c009b30).stop 2021-11-20T07:42:59.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 shutdown_connections 2021-11-20T07:42:59.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f8d3405c300 0x7f8d3405e7c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.767 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d4c0f4820 0x7f8d4c0f73d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 --2- 172.21.15.71:0/2796914581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d4c0f51c0 0x7f8d4c0f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:42:59.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 >> 172.21.15.71:0/2796914581 conn(0x7f8d4c0f0320 msgr2=0x7f8d4c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:42:59.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 shutdown_connections 2021-11-20T07:42:59.768 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:42:59.763+0000 7f8d4bfff700 1 -- 172.21.15.71:0/2796914581 wait complete. 2021-11-20T07:42:59.768 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2021-11-20T07:42:59.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:42:49.432478+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24451, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 78, 'state': 'up:clientreplay', 'state_seq': 27, 'addr': '172.21.15.112:6827/845332301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 845332301}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 845332301}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:42:59.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24451:up:clientreplay 2021-11-20T07:43:00.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:59 smithi071 conmon[33687]: audit 2021-11-20T07:42:59.762950+0000 mon.smithi071 (mon.0 2021-11-20T07:43:00.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:42:59 smithi071 conmon[33687]: ) 894 : audit [DBG] from='client.? 172.21.15.71:0/2796914581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:00.157 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:59 smithi112 conmon[36062]: audit 2021-11-20T07:42:59.762950+0000 mon.smithi071 (mon 2021-11-20T07:43:00.157 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:42:59 smithi112 conmon[36062]: .0) 894 : audit [DBG] from='client.? 172.21.15.71:0/2796914581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:00.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:00.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.903+0000 7f3d2fa18700 1 -- 172.21.15.71:0/606741326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f6c60 msgr2=0x7f3d280f7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:00.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.903+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/606741326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f6c60 0x7f3d280f7080 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f3d24004660 tx=0x7f3d24009b30).stop 2021-11-20T07:43:00.909 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- 172.21.15.71:0/606741326 shutdown_connections 2021-11-20T07:43:00.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/606741326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f5970 0x7f3d280f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:00.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/606741326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f6c60 0x7f3d280f7080 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:00.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- 172.21.15.71:0/606741326 >> 172.21.15.71:0/606741326 conn(0x7f3d280f04c0 msgr2=0x7f3d280f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:00.910 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- 172.21.15.71:0/606741326 shutdown_connections 2021-11-20T07:43:00.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- 172.21.15.71:0/606741326 wait complete. 2021-11-20T07:43:00.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 Processor -- start 2021-11-20T07:43:00.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- start start 2021-11-20T07:43:00.911 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f6c60 0x7f3d281073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28107970 con 0x7f3d280f5970 2021-11-20T07:43:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2fa18700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28107ab0 con 0x7f3d280f6c60 2021-11-20T07:43:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2d7b4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:00.912 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2d7b4700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49750/0 (socket says 172.21.15.71:49750) 2021-11-20T07:43:00.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.904+0000 7f3d2d7b4700 1 -- 172.21.15.71:0/3313131730 learned_addr learned my addr 172.21.15.71:0/3313131730 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:00.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2cfb3700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f6c60 0x7f3d281073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:00.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2d7b4700 1 -- 172.21.15.71:0/3313131730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f6c60 msgr2=0x7f3d281073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:00.913 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2d7b4700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f6c60 0x7f3d281073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:00.914 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2d7b4700 1 -- 172.21.15.71:0/3313131730 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d24005040 con 0x7f3d280f5970 2021-11-20T07:43:00.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2d7b4700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3d24009f50 tx=0x7f3d24009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:00.917 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d24018070 con 0x7f3d280f5970 2021-11-20T07:43:00.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2810c4f0 con 0x7f3d280f5970 2021-11-20T07:43:00.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d24007ba0 con 0x7f3d280f5970 2021-11-20T07:43:00.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d2401c3c0 con 0x7f3d280f5970 2021-11-20T07:43:00.918 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.905+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2810cab0 con 0x7f3d280f5970 2021-11-20T07:43:00.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.906+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d280fde10 con 0x7f3d280f5970 2021-11-20T07:43:00.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.906+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3d2401d420 con 0x7f3d280f5970 2021-11-20T07:43:00.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.907+0000 7f3d1dffb700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 0x7f3d1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:00.919 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.907+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f3d240781e0 con 0x7f3d280f5970 2021-11-20T07:43:00.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.907+0000 7f3d2cfb3700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 0x7f3d1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:00.920 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.907+0000 7f3d2cfb3700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 0x7f3d1405e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3d180096c0 tx=0x7f3d18009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:00.921 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:00.911+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2404a8f0 con 0x7f3d280f5970 2021-11-20T07:43:01.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021-11-20T07:43:00.282552+0000 mon.smithi071 (mon.0 2021-11-20T07:43:01.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: ) 895 : cluster [INF] daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:43:01.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021-11-20T07:43:00.286735+0000 mon.smithi071 (mon.0) 896 : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T07:43:01.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021-11-20T07:43:00.420467+0000 mon.smithi071 (mon.0) 2021-11-20T07:43:01.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: 897 : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:43:01.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021-11-20T07:43 2021-11-20T07:43:01.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: :00.423805+0000 mon.smithi071 (mon.0) 898 : 2021-11-20T07:43:01.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster [DBG] mds.? [v2:172.21.15.112:6826/503879816,v1:172.21.15.112:6827/503879816] up:boot 2021-11-20T07:43:01.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021-11-20T07:43: 2021-11-20T07:43:01.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: 00.423884+0000 mon.smithi071 (mon.0) 899 : cluster 2021-11-20T07:43:01.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:43:01.070 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:43:01.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: -20T07:43:00.424231+0000 mon.smithi071 (mon 2021-11-20T07:43:01.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: .0) 900 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:43:01.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: cluster 2021 2021-11-20T07:43:01.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: -11-20T07:43:00.428836+0000 2021-11-20T07:43:01.072 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:00 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 331 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 307 B/s wr, 0 op/s 2021-11-20T07:43:01.097 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.095+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d2810d1b0 con 0x7f3d280f5970 2021-11-20T07:43:01.098 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.095+0000 7f3d1dffb700 1 -- 172.21.15.71:0/3313131730 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f3d2810d1b0 con 0x7f3d280f5970 2021-11-20T07:43:01.099 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:01.099 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:00.420549+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:replay","state_seq":1,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.096+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 msgr2=0x7f3d1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:01.102 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 0x7f3d1405e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3d180096c0 tx=0x7f3d18009be0).stop 2021-11-20T07:43:01.103 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 msgr2=0x7f3d28106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:01.103 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3d24009f50 tx=0x7f3d24009b30).stop 2021-11-20T07:43:01.103 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 shutdown_connections 2021-11-20T07:43:01.104 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3d1405c2a0 0x7f3d1405e760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:01.104 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d280f5970 0x7f3d28106ed0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:01.105 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 --2- 172.21.15.71:0/3313131730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d280f6c60 0x7f3d281073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:01.105 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 >> 172.21.15.71:0/3313131730 conn(0x7f3d280f04c0 msgr2=0x7f3d280f5040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:01.106 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 shutdown_connections 2021-11-20T07:43:01.106 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:01.097+0000 7f3d2fa18700 1 -- 172.21.15.71:0/3313131730 wait complete. 2021-11-20T07:43:01.106 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 85 2021-11-20T07:43:01.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:00.420549+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:01.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:replay 2021-11-20T07:43:01.203 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021-11-20T07:43:00.282552+0000 mon.smithi071 (mon.0) 895 : cluster [INF] daemon mds.cephfs.smithi112.iqrmps restarted 2021-11-20T07:43:01.204 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021-11-20T07:43:00.286735+0000 mon.smithi071 (mon.0) 896 : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T07:43:01.204 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021-11-20T07:43:00.420467+0000 mon.smithi071 (mon 2021-11-20T07:43:01.204 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: .0) 897 : cluster [INF] Standby daemon mds.cephfs.smithi112.iqrmps assigned to filesystem cephfs as rank 0 2021-11-20T07:43:01.204 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021-11-20T07:43:01.205 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: 2021-11-20T07:43:00.423805+0000 mon.smithi071 (mon. 2021-11-20T07:43:01.205 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: 0) 898 : cluster [DBG] mds.? [v2:172.21.15.112:6826/503879816,v1:172.21.15.112:6827/503879816] up:boot 2021-11-20T07:43:01.205 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021-11-20 2021-11-20T07:43:01.205 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: T07:43:00.423884+0000 mon.smithi071 (mon 2021-11-20T07:43:01.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: .0) 899 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:replay,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:43:01.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:43:01.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: -20T07:43:00.424231+0000 mon.smithi071 2021-11-20T07:43:01.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: (mon.0) 900 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.iqrmps"}]: dispatch 2021-11-20T07:43:01.207 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: cluster 2021 2021-11-20T07:43:01.208 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: -11-20T07:43:00.428836+0000 2021-11-20T07:43:01.208 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: mgr.smithi071.ynqmwn (mgr.14162) 331 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 307 B/s wr, 0 op/s 2021-11-20T07:43:01.208 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:00 smithi112 conmon[36062]: 2021-11-20T07:43:02.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:02.136 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:01 smithi071 conmon[33687]: audit 2021-11-20T07:43:01.096428+0000 2021-11-20T07:43:02.137 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:01 smithi071 conmon[33687]: mon.smithi071 (mon.0) 901 : audit [DBG] from='client.? 172.21.15.71:0/3313131730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:02.138 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:01 smithi071 conmon[33687]: cluster 2021-11-20T07:43:01.432012+0000 mon.smithi071 (mon.0) 902 : cluster [DBG] mds.? [v2:172.21.15.112:6826/503879816,v1:172.21.15.112:6827/503879816] up:resolve 2021-11-20T07:43:02.138 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:01 smithi071 conmon[33687]: cluster 2021-11-20T07 2021-11-20T07:43:02.138 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:01 smithi071 conmon[33687]: :43:01.432088+0000 mon.smithi071 (mon.0) 903 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:43:02.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 -- 172.21.15.71:0/1112349979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 msgr2=0x7f75400f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:02.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 --2- 172.21.15.71:0/1112349979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f75400f6b60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f7534004660 tx=0x7f7534009b30).stop 2021-11-20T07:43:02.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 -- 172.21.15.71:0/1112349979 shutdown_connections 2021-11-20T07:43:02.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 --2- 172.21.15.71:0/1112349979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f75400f6b60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 --2- 172.21.15.71:0/1112349979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 0x7f75400f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 -- 172.21.15.71:0/1112349979 >> 172.21.15.71:0/1112349979 conn(0x7f75400f0ba0 msgr2=0x7f75400f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:02.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 -- 172.21.15.71:0/1112349979 shutdown_connections 2021-11-20T07:43:02.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.226+0000 7f754541f700 1 -- 172.21.15.71:0/1112349979 wait complete. 2021-11-20T07:43:02.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 Processor -- start 2021-11-20T07:43:02.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 -- start start 2021-11-20T07:43:02.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 0x7f7540106d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:02.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:02.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7540107870 con 0x7f75400f6700 2021-11-20T07:43:02.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f754541f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7540108080 con 0x7f75400f5460 2021-11-20T07:43:02.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:02.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49764/0 (socket says 172.21.15.71:49764) 2021-11-20T07:43:02.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 -- 172.21.15.71:0/1545931015 learned_addr learned my addr 172.21.15.71:0/1545931015 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:02.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753ffff700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 0x7f7540106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:02.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 -- 172.21.15.71:0/1545931015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 msgr2=0x7f7540106d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:02.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 0x7f7540106d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 -- 172.21.15.71:0/1545931015 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7534005040 con 0x7f75400f6700 2021-11-20T07:43:02.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753f7fe700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7534009830 tx=0x7f7534009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:02.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.227+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7534018070 con 0x7f75400f6700 2021-11-20T07:43:02.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.228+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75401082a0 con 0x7f75400f6700 2021-11-20T07:43:02.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.228+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7534013410 con 0x7f75400f6700 2021-11-20T07:43:02.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.228+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f753401c620 con 0x7f75400f6700 2021-11-20T07:43:02.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.228+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7540108800 con 0x7f75400f6700 2021-11-20T07:43:02.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.229+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75400f4850 con 0x7f75400f6700 2021-11-20T07:43:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.229+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f753401c8e0 con 0x7f75400f6700 2021-11-20T07:43:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.229+0000 7f753cff9700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 0x7f752805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.229+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f7534078350 con 0x7f75400f6700 2021-11-20T07:43:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.229+0000 7f753ffff700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 0x7f752805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:02.242 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.230+0000 7f753ffff700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 0x7f752805e7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f7540106af0 tx=0x7f7530009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.233+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f753404aa00 con 0x7f75400f6700 2021-11-20T07:43:02.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: audit 2021-11-20T07:43:01.096428+0000 mon.smithi071 (mon.0) 901 2021-11-20T07:43:02.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/3313131730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:02.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: cluster 2021-11-20T07:43:01.432012+0000 mon.smithi071 (mon.0) 902 : cluster [DBG] mds.? [v2:172.21.15.112:6826/503879816,v1:172.21.15.112:6827/503879816] up:resolve 2021-11-20T07:43:02.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: cluster 2021-11-20T07:43:01. 2021-11-20T07:43:02.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: 432088+0000 mon.smithi071 (mon.0) 903 : cluster [DBG] 2021-11-20T07:43:02.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:01 smithi112 conmon[36062]: fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:active} 2021-11-20T07:43:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.418+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7540045b20 con 0x7f75400f6700 2021-11-20T07:43:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.419+0000 7f753cff9700 1 -- 172.21.15.71:0/1545931015 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f753404e020 con 0x7f75400f6700 2021-11-20T07:43:02.421 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:02.422 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:02.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.420+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 msgr2=0x7f752805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:02.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.420+0000 7f754541f700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 0x7f752805e7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f7540106af0 tx=0x7f7530009040).stop 2021-11-20T07:43:02.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.420+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 msgr2=0x7f7540107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:02.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.420+0000 7f754541f700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7534009830 tx=0x7f7534009b30).stop 2021-11-20T07:43:02.424 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 shutdown_connections 2021-11-20T07:43:02.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75400f5460 0x7f7540106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f752805c300 0x7f752805e7c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 --2- 172.21.15.71:0/1545931015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75400f6700 0x7f7540107270 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:02.425 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 >> 172.21.15.71:0/1545931015 conn(0x7f75400f0ba0 msgr2=0x7f754005fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:02.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 shutdown_connections 2021-11-20T07:43:02.426 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:02.421+0000 7f754541f700 1 -- 172.21.15.71:0/1545931015 wait complete. 2021-11-20T07:43:02.426 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:02.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:02.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:03.194 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:02 smithi071 conmon[33687]: audit 2021-11-20T07:43:02.420392+0000 mon.smithi071 (mon.0) 2021-11-20T07:43:03.195 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:02 smithi071 conmon[33687]: 904 : audit [DBG] from='client.? 172.21.15.71:0/1545931015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:03.195 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:02 smithi071 conmon[33687]: cluster 2021-11-20T07:43:02.429291+0000 mgr.smithi071.ynqmwn (mgr.14162) 332 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:43:03.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:02 smithi112 conmon[36062]: audit 2021-11-20T07:43:02.420392+0000 mon.smithi071 (mon.0) 904 : audit 2021-11-20T07:43:03.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:02 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/1545931015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:03.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:02 smithi112 conmon[36062]: cluster 2021-11-20T07:43:02.429291+0000 mgr.smithi071.ynqmwn (mgr.14162) 332 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 B/s wr, 0 op/s 2021-11-20T07:43:03.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 -- 172.21.15.71:0/97488484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f4780 msgr2=0x7f3fbc0f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 --2- 172.21.15.71:0/97488484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f4c00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f3fb0004660 tx=0x7f3fb0009b30).stop 2021-11-20T07:43:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 -- 172.21.15.71:0/97488484 shutdown_connections 2021-11-20T07:43:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 --2- 172.21.15.71:0/97488484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f4c00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 --2- 172.21.15.71:0/97488484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.558+0000 7f3fc0828700 1 -- 172.21.15.71:0/97488484 >> 172.21.15.71:0/97488484 conn(0x7f3fbc0f04a0 msgr2=0x7f3fbc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 -- 172.21.15.71:0/97488484 shutdown_connections 2021-11-20T07:43:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 -- 172.21.15.71:0/97488484 wait complete. 2021-11-20T07:43:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 Processor -- start 2021-11-20T07:43:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 -- start start 2021-11-20T07:43:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0f7900 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fc0828700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0f5ea0 con 0x7f3fbc0f4780 2021-11-20T07:43:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fb9d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fb9d9b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49776/0 (socket says 172.21.15.71:49776) 2021-11-20T07:43:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fb9d9b700 1 -- 172.21.15.71:0/474642937 learned_addr learned my addr 172.21.15.71:0/474642937 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.559+0000 7f3fba59c700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fb9d9b700 1 -- 172.21.15.71:0/474642937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f4780 msgr2=0x7f3fbc0f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fb9d9b700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fb9d9b700 1 -- 172.21.15.71:0/474642937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0005040 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fb9d9b700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3fb0000c00 tx=0x7f3fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb0019070 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc0f60c0 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb0007e70 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb0014d40 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.560+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0f6620 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.561+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc046330 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.561+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3fb0025070 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.561+0000 7f3faaffd700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 0x7f3fa405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.562+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f3fb0077400 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.562+0000 7f3fba59c700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 0x7f3fa405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.562+0000 7f3fba59c700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 0x7f3fa405e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3fac009fd0 tx=0x7f3fac00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:03.575 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.565+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fb0049b10 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.749+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fbc027a00 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.750+0000 7f3faaffd700 1 -- 172.21.15.71:0/474642937 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f3fb004d130 con 0x7f3fbc0f7d90 2021-11-20T07:43:03.752 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:03.753 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:03.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 msgr2=0x7f3fa405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:03.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 0x7f3fa405e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3fac009fd0 tx=0x7f3fac00b040).stop 2021-11-20T07:43:03.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 msgr2=0x7f3fbc0f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:03.755 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3fb0000c00 tx=0x7f3fb000b040).stop 2021-11-20T07:43:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 shutdown_connections 2021-11-20T07:43:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3fbc0f4780 0x7f3fbc0f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f3fa405c2a0 0x7f3fa405e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 --2- 172.21.15.71:0/474642937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fbc0f7d90 0x7f3fbc0f59b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:03.757 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.751+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 >> 172.21.15.71:0/474642937 conn(0x7f3fbc0f04a0 msgr2=0x7f3fbc0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:03.757 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.752+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 shutdown_connections 2021-11-20T07:43:03.757 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:03.752+0000 7f3fc0828700 1 -- 172.21.15.71:0/474642937 wait complete. 2021-11-20T07:43:03.757 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:03.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:03.766 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:04.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:03 smithi071 conmon[33687]: audit 2021-11-20T07:43:03.751287+0000 mon.smithi071 (mon.0) 905 : audit [DBG] from='client.? 172.21.15.71:0/474642937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:04.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:03 smithi112 conmon[36062]: audit 2021-11-20T07:43:03.751287+0000 mon.smithi071 (mon.0) 905 2021-11-20T07:43:04.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:03 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/474642937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:04.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:04.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 -- 172.21.15.71:0/4122620083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 msgr2=0x7f31e80f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:04.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/4122620083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e80f6aa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f31d8004660 tx=0x7f31d8009b30).stop 2021-11-20T07:43:04.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 -- 172.21.15.71:0/4122620083 shutdown_connections 2021-11-20T07:43:04.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/4122620083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e80f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:04.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/4122620083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e80f6aa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:04.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.892+0000 7f31ed8c2700 1 -- 172.21.15.71:0/4122620083 >> 172.21.15.71:0/4122620083 conn(0x7f31e80f1de0 msgr2=0x7f31e80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:04.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 -- 172.21.15.71:0/4122620083 shutdown_connections 2021-11-20T07:43:04.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 -- 172.21.15.71:0/4122620083 wait complete. 2021-11-20T07:43:04.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 Processor -- start 2021-11-20T07:43:04.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 -- start start 2021-11-20T07:43:04.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:04.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e81073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:04.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e8107950 con 0x7f31e80f6680 2021-11-20T07:43:04.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31ed8c2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e8107a90 con 0x7f31e80f7920 2021-11-20T07:43:04.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31e6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:04.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31e6ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49784/0 (socket says 172.21.15.71:49784) 2021-11-20T07:43:04.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.893+0000 7f31e6ffd700 1 -- 172.21.15.71:0/3846725370 learned_addr learned my addr 172.21.15.71:0/3846725370 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:04.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31dffff700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e81073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:04.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31e6ffd700 1 -- 172.21.15.71:0/3846725370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 msgr2=0x7f31e81073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:04.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31e6ffd700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e81073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31e6ffd700 1 -- 172.21.15.71:0/3846725370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d8005040 con 0x7f31e80f6680 2021-11-20T07:43:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31dffff700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e81073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:43:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31e6ffd700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f31d8009f50 tx=0x7f31d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d8018070 con 0x7f31e80f6680 2021-11-20T07:43:04.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31d8013410 con 0x7f31e80f6680 2021-11-20T07:43:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d801c620 con 0x7f31e80f6680 2021-11-20T07:43:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e810c4d0 con 0x7f31e80f6680 2021-11-20T07:43:04.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.894+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e810ca90 con 0x7f31e80f6680 2021-11-20T07:43:04.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.895+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e8027a00 con 0x7f31e80f6680 2021-11-20T07:43:04.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.895+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f31d801d420 con 0x7f31e80f6680 2021-11-20T07:43:04.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.895+0000 7f31df7fe700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 0x7f31d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:04.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.896+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f31d80782e0 con 0x7f31e80f6680 2021-11-20T07:43:04.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.896+0000 7f31dffff700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 0x7f31d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:04.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.896+0000 7f31dffff700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 0x7f31d405e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f31d0004a70 tx=0x7f31d0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:04.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:04.899+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31d804a9f0 con 0x7f31e80f6680 2021-11-20T07:43:05.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:04 smithi071 conmon[33687]: cluster 2021-11-20T07:43:04.429739+0000 mgr.smithi071.ynqmwn (mgr.14162) 333 : cluster 2021-11-20T07:43:05.068 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:04 smithi071 conmon[33687]: [DBG] pgmap v311: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:05.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.085+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31e80fee10 con 0x7f31e80f6680 2021-11-20T07:43:05.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.086+0000 7f31df7fe700 1 -- 172.21.15.71:0/3846725370 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f31d804e010 con 0x7f31e80f6680 2021-11-20T07:43:05.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:05.088 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 msgr2=0x7f31d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 0x7f31d405e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f31d0004a70 tx=0x7f31d0009040).stop 2021-11-20T07:43:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 msgr2=0x7f31e8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f31d8009f50 tx=0x7f31d8009b30).stop 2021-11-20T07:43:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 shutdown_connections 2021-11-20T07:43:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f31d405c2a0 0x7f31d405e760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:05.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f31e80f6680 0x7f31e8106eb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:05.091 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 --2- 172.21.15.71:0/3846725370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31e80f7920 0x7f31e81073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 >> 172.21.15.71:0/3846725370 conn(0x7f31e80f1de0 msgr2=0x7f31e80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 shutdown_connections 2021-11-20T07:43:05.092 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:05.087+0000 7f31ed8c2700 1 -- 172.21.15.71:0/3846725370 wait complete. 2021-11-20T07:43:05.093 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:05.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:05.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:04 smithi112 conmon[36062]: cluster 2021-11-20T07:43:04.429739+0000 mgr.smithi071.ynqmwn (mgr.14162) 333 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:06.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:06.131 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:05 smithi071 conmon[33687]: audit 2021-11-20T07:43:05.087151+0000 mon.smithi071 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.71:0/3846725370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:06.132 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:05 smithi071 conmon[33687]: 2021-11-20T07:43:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 -- 172.21.15.71:0/739119208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 msgr2=0x7fc5980f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/739119208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc5980f69f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fc588004660 tx=0x7fc588009b30).stop 2021-11-20T07:43:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 -- 172.21.15.71:0/739119208 shutdown_connections 2021-11-20T07:43:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/739119208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5980f7870 0x7fc5980f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/739119208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc5980f69f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 -- 172.21.15.71:0/739119208 >> 172.21.15.71:0/739119208 conn(0x7fc5980f1d10 msgr2=0x7fc5980f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 -- 172.21.15.71:0/739119208 shutdown_connections 2021-11-20T07:43:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.227+0000 7fc59d3a3700 1 -- 172.21.15.71:0/739119208 wait complete. 2021-11-20T07:43:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 Processor -- start 2021-11-20T07:43:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 -- start start 2021-11-20T07:43:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5980f7870 0x7fc598069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806a380 con 0x7fc5980f65d0 2021-11-20T07:43:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc59d3a3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806a4c0 con 0x7fc5980f7870 2021-11-20T07:43:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49796/0 (socket says 172.21.15.71:49796) 2021-11-20T07:43:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 -- 172.21.15.71:0/226965753 learned_addr learned my addr 172.21.15.71:0/226965753 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 -- 172.21.15.71:0/226965753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5980f7870 msgr2=0x7fc598069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5980f7870 0x7fc598069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.228+0000 7fc597fff700 1 -- 172.21.15.71:0/226965753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588005040 con 0x7fc5980f65d0 2021-11-20T07:43:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc597fff700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc588009f50 tx=0x7fc588009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc588018070 con 0x7fc5980f65d0 2021-11-20T07:43:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc588013410 con 0x7fc5980f65d0 2021-11-20T07:43:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59806a740 con 0x7fc5980f65d0 2021-11-20T07:43:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc58801c620 con 0x7fc5980f65d0 2021-11-20T07:43:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.229+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59806aca0 con 0x7fc5980f65d0 2021-11-20T07:43:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.230+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc598045ad0 con 0x7fc5980f65d0 2021-11-20T07:43:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.230+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc58801d420 con 0x7fc5980f65d0 2021-11-20T07:43:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.230+0000 7fc594ff9700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 0x7fc58005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.231+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fc588078240 con 0x7fc5980f65d0 2021-11-20T07:43:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.231+0000 7fc5977fe700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 0x7fc58005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.231+0000 7fc5977fe700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 0x7fc58005e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc58c004880 tx=0x7fc58c007ae0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.235+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc58804a9d0 con 0x7fc5980f65d0 2021-11-20T07:43:06.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:05 smithi112 conmon[36062]: audit 2021-11-20T07:43:05.087151+0000 mon.smithi071 (mon 2021-11-20T07:43:06.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:05 smithi112 conmon[36062]: .0) 906 : audit [DBG] from='client.? 172.21.15.71:0/3846725370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:06.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.416+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5980fee30 con 0x7fc5980f65d0 2021-11-20T07:43:06.419 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.417+0000 7fc594ff9700 1 -- 172.21.15.71:0/226965753 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7fc58804dff0 con 0x7fc5980f65d0 2021-11-20T07:43:06.419 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:06.419 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 msgr2=0x7fc58005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 0x7fc58005e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc58c004880 tx=0x7fc58c007ae0).stop 2021-11-20T07:43:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 msgr2=0x7fc59806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc588009f50 tx=0x7fc588009b30).stop 2021-11-20T07:43:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 shutdown_connections 2021-11-20T07:43:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fc58005c300 0x7fc58005e7c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5980f65d0 0x7fc59806b9a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 --2- 172.21.15.71:0/226965753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5980f7870 0x7fc598069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 >> 172.21.15.71:0/226965753 conn(0x7fc5980f1d10 msgr2=0x7fc5980fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 shutdown_connections 2021-11-20T07:43:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:06.418+0000 7fc59d3a3700 1 -- 172.21.15.71:0/226965753 wait complete. 2021-11-20T07:43:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:06.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:06.432 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:07.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:06 smithi112 conmon[36062]: audit 2021-11-20T07:43:06.418104+0000 mon.smithi071 (mon.0) 907 : audit 2021-11-20T07:43:07.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:06 smithi112 conmon[36062]: [DBG] from='client.? 172.21.15.71:0/226965753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:07.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:06 smithi112 conmon[36062]: cluster 2021-11-20T07:43:06.430165+0000 mgr.smithi071.ynqmwn (mgr.14162) 334 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:07.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:06 smithi071 conmon[33687]: audit 2021-11-20T07:43:06.418104+0000 mon.smithi071 (mon.0) 2021-11-20T07:43:07.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:06 smithi071 conmon[33687]: 907 : audit [DBG] from='client.? 172.21.15.71:0/226965753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:07.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:06 smithi071 conmon[33687]: cluster 2021-11-20T07:43:06.430165+0000 mgr.smithi071.ynqmwn (mgr.14162) 334 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:07.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:07.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- 172.21.15.71:0/3008925399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f5fc0 msgr2=0x7fa2480f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:07.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 --2- 172.21.15.71:0/3008925399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f5fc0 0x7fa2480f6420 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fa2300099e0 tx=0x7fa230004d10).stop 2021-11-20T07:43:07.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- 172.21.15.71:0/3008925399 shutdown_connections 2021-11-20T07:43:07.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 --2- 172.21.15.71:0/3008925399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f5fc0 0x7fa2480f6420 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 --2- 172.21.15.71:0/3008925399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f4d20 0x7fa2480f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- 172.21.15.71:0/3008925399 >> 172.21.15.71:0/3008925399 conn(0x7fa2480f0440 msgr2=0x7fa2480f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:07.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- 172.21.15.71:0/3008925399 shutdown_connections 2021-11-20T07:43:07.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- 172.21.15.71:0/3008925399 wait complete. 2021-11-20T07:43:07.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 Processor -- start 2021-11-20T07:43:07.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.558+0000 7fa24f03c700 1 -- start start 2021-11-20T07:43:07.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24f03c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:07.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24f03c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 0x7fa24810bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:07.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24f03c700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24810c2b0 con 0x7fa2480f4d20 2021-11-20T07:43:07.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24f03c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24810c3f0 con 0x7fa2480f5fc0 2021-11-20T07:43:07.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:07.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49806/0 (socket says 172.21.15.71:49806) 2021-11-20T07:43:07.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 -- 172.21.15.71:0/1036222623 learned_addr learned my addr 172.21.15.71:0/1036222623 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:07.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa247fff700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 0x7fa24810bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:07.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 -- 172.21.15.71:0/1036222623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 msgr2=0x7fa24810bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:07.570 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 0x7fa24810bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 -- 172.21.15.71:0/1036222623 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa230009640 con 0x7fa2480f4d20 2021-11-20T07:43:07.571 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa247fff700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 0x7fa24810bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T07:43:07.572 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa24cdd8700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fa238002730 tx=0x7fa238009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:07.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.559+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa238009320 con 0x7fa2480f4d20 2021-11-20T07:43:07.573 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.560+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24810c650 con 0x7fa2480f4d20 2021-11-20T07:43:07.574 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.560+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa23800f040 con 0x7fa2480f4d20 2021-11-20T07:43:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.560+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa238014840 con 0x7fa2480f4d20 2021-11-20T07:43:07.575 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.560+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24810cb90 con 0x7fa2480f4d20 2021-11-20T07:43:07.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.560+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2480fd4b0 con 0x7fa2480f4d20 2021-11-20T07:43:07.576 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.561+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa238009480 con 0x7fa2480f4d20 2021-11-20T07:43:07.577 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.561+0000 7fa2457fa700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 0x7fa22805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:07.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.561+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fa238076850 con 0x7fa2480f4d20 2021-11-20T07:43:07.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.561+0000 7fa247fff700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 0x7fa22805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:07.578 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.562+0000 7fa247fff700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 0x7fa22805e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa23000b620 tx=0x7fa230004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:07.579 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.565+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa238048f60 con 0x7fa2480f4d20 2021-11-20T07:43:07.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.747+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa24810d630 con 0x7fa2480f4d20 2021-11-20T07:43:07.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.748+0000 7fa2457fa700 1 -- 172.21.15.71:0/1036222623 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7fa23804c580 con 0x7fa2480f4d20 2021-11-20T07:43:07.750 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:07.751 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 msgr2=0x7fa22805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 0x7fa22805e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa23000b620 tx=0x7fa230004d10).stop 2021-11-20T07:43:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 msgr2=0x7fa2480f7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fa238002730 tx=0x7fa238009d80).stop 2021-11-20T07:43:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 shutdown_connections 2021-11-20T07:43:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7fa22805c2a0 0x7fa22805e760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa2480f4d20 0x7fa2480f7cb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 --2- 172.21.15.71:0/1036222623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2480f5fc0 0x7fa24810bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.749+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 >> 172.21.15.71:0/1036222623 conn(0x7fa2480f0440 msgr2=0x7fa2480fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.750+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 shutdown_connections 2021-11-20T07:43:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:07.750+0000 7fa24f03c700 1 -- 172.21.15.71:0/1036222623 wait complete. 2021-11-20T07:43:07.755 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:07.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:07.763 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:08.069 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:07 smithi071 conmon[33687]: audit 2021-11-20T07:43:07.749180+0000 mon.smithi071 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.71:0/1036222623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:08.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:07 smithi112 conmon[36062]: audit 2021-11-20T07:43:07.749180+0000 mon.smithi071 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.71:0/1036222623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:08.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:07 smithi112 conmon[36062]: 2021-11-20T07:43:08.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:08.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- 172.21.15.71:0/1734904556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f4780 msgr2=0x7f9dd00f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:08.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1734904556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f4780 0x7f9dd00f4c00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f9dc0004660 tx=0x7f9dc0009b30).stop 2021-11-20T07:43:08.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- 172.21.15.71:0/1734904556 shutdown_connections 2021-11-20T07:43:08.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1734904556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f4780 0x7f9dd00f4c00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:08.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1734904556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9dd00f5b00 0x7f9dd00f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:08.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- 172.21.15.71:0/1734904556 >> 172.21.15.71:0/1734904556 conn(0x7f9dd00f04a0 msgr2=0x7f9dd00f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:08.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- 172.21.15.71:0/1734904556 shutdown_connections 2021-11-20T07:43:08.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- 172.21.15.71:0/1734904556 wait complete. 2021-11-20T07:43:08.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 Processor -- start 2021-11-20T07:43:08.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- start start 2021-11-20T07:43:08.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9dd00f4780 0x7f9dd010d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:08.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:08.899 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd010e090 con 0x7f9dd00f5b00 2021-11-20T07:43:08.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd8088700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd010e1d0 con 0x7f9dd00f4780 2021-11-20T07:43:08.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd5623700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:08.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd5623700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49818/0 (socket says 172.21.15.71:49818) 2021-11-20T07:43:08.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.892+0000 7f9dd5623700 1 -- 172.21.15.71:0/1604589260 learned_addr learned my addr 172.21.15.71:0/1604589260 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:08.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd5623700 1 -- 172.21.15.71:0/1604589260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9dd00f4780 msgr2=0x7f9dd010d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:08.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd5623700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9dd00f4780 0x7f9dd010d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:08.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd5623700 1 -- 172.21.15.71:0/1604589260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0005040 con 0x7f9dd00f5b00 2021-11-20T07:43:08.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd5623700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f9dc0009830 tx=0x7f9dc0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:08.902 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc0018070 con 0x7f9dd00f5b00 2021-11-20T07:43:08.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0112c10 con 0x7f9dd00f5b00 2021-11-20T07:43:08.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dc0013410 con 0x7f9dd00f5b00 2021-11-20T07:43:08.903 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc001c620 con 0x7f9dd00f5b00 2021-11-20T07:43:08.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.893+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd0113170 con 0x7f9dd00f5b00 2021-11-20T07:43:08.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.894+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd0027a00 con 0x7f9dd00f5b00 2021-11-20T07:43:08.904 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.894+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9dc001c8e0 con 0x7f9dd00f5b00 2021-11-20T07:43:08.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.895+0000 7f9dc67fc700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 0x7f9dbc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:08.905 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.895+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f9dc0078310 con 0x7f9dd00f5b00 2021-11-20T07:43:08.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.895+0000 7f9dd5e24700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 0x7f9dbc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:08.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.895+0000 7f9dd5e24700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 0x7f9dbc05e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f9dc8006fd0 tx=0x7f9dc8006e30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:08.906 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:08.899+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dc004aa20 con 0x7f9dd00f5b00 2021-11-20T07:43:09.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:08 smithi071 conmon[33687]: cluster 2021-11-20T07:43:08.430554+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:43:09.071 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:08 smithi071 conmon[33687]: 335 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:09.084 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.082+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd01055a0 con 0x7f9dd00f5b00 2021-11-20T07:43:09.085 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.082+0000 7f9dc67fc700 1 -- 172.21.15.71:0/1604589260 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f9dc004e040 con 0x7f9dd00f5b00 2021-11-20T07:43:09.085 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:09.086 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:09.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 msgr2=0x7f9dbc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:09.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 0x7f9dbc05e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f9dc8006fd0 tx=0x7f9dc8006e30).stop 2021-11-20T07:43:09.087 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 msgr2=0x7f9dd010da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:09.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f9dc0009830 tx=0x7f9dc0009b30).stop 2021-11-20T07:43:09.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 shutdown_connections 2021-11-20T07:43:09.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9dd00f4780 0x7f9dd010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:09.088 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f9dbc05c2a0 0x7f9dbc05e760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:09.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 --2- 172.21.15.71:0/1604589260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9dd00f5b00 0x7f9dd010da50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:09.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 >> 172.21.15.71:0/1604589260 conn(0x7f9dd00f04a0 msgr2=0x7f9dd00f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:09.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 shutdown_connections 2021-11-20T07:43:09.089 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:09.084+0000 7f9dd8088700 1 -- 172.21.15.71:0/1604589260 wait complete. 2021-11-20T07:43:09.090 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:09.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:09.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:09.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:08 smithi112 conmon[36062]: cluster 2021-11-20T07:43:08.430554+0000 mgr.smithi071.ynqmwn (mgr.14162) 335 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:10.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:10.127 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:09 smithi071 conmon[33687]: audit 2021-11-20T07:43:09.083664+0000 mon.smithi071 (mon.0) 909 : audit [DBG] from='client.? 172.21.15.71:0/1604589260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:10.127 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:09 smithi071 conmon[33687]: 2021-11-20T07:43:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 -- 172.21.15.71:0/2393407417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f7870 msgr2=0x7ff6540f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 --2- 172.21.15.71:0/2393407417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f7870 0x7ff6540f7cd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7ff640004660 tx=0x7ff640009b30).stop 2021-11-20T07:43:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 -- 172.21.15.71:0/2393407417 shutdown_connections 2021-11-20T07:43:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 --2- 172.21.15.71:0/2393407417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f7870 0x7ff6540f7cd0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 --2- 172.21.15.71:0/2393407417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f65d0 0x7ff6540f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 -- 172.21.15.71:0/2393407417 >> 172.21.15.71:0/2393407417 conn(0x7ff6540f1d10 msgr2=0x7ff6540f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 -- 172.21.15.71:0/2393407417 shutdown_connections 2021-11-20T07:43:10.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.223+0000 7ff659fc0700 1 -- 172.21.15.71:0/2393407417 wait complete. 2021-11-20T07:43:10.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 Processor -- start 2021-11-20T07:43:10.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 -- start start 2021-11-20T07:43:10.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:10.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f7870 0x7ff654107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:10.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654107820 con 0x7ff6540f65d0 2021-11-20T07:43:10.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff659fc0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654108030 con 0x7ff6540f7870 2021-11-20T07:43:10.232 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff658fbe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:10.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff658fbe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49828/0 (socket says 172.21.15.71:49828) 2021-11-20T07:43:10.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.224+0000 7ff658fbe700 1 -- 172.21.15.71:0/4198452086 learned_addr learned my addr 172.21.15.71:0/4198452086 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:10.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff653fff700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f7870 0x7ff654107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:10.233 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff658fbe700 1 -- 172.21.15.71:0/4198452086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f7870 msgr2=0x7ff654107220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:10.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff658fbe700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f7870 0x7ff654107220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff658fbe700 1 -- 172.21.15.71:0/4198452086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640005040 con 0x7ff6540f65d0 2021-11-20T07:43:10.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff658fbe700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ff648002700 tx=0x7ff648009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:10.234 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff648009320 con 0x7ff6540f65d0 2021-11-20T07:43:10.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff64800f040 con 0x7ff6540f65d0 2021-11-20T07:43:10.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6541082b0 con 0x7ff6540f65d0 2021-11-20T07:43:10.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6480148e0 con 0x7ff6540f65d0 2021-11-20T07:43:10.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.225+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff654108870 con 0x7ff6540f65d0 2021-11-20T07:43:10.235 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.226+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6540f59c0 con 0x7ff6540f65d0 2021-11-20T07:43:10.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.226+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff648014a40 con 0x7ff6540f65d0 2021-11-20T07:43:10.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.226+0000 7ff6517fa700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 0x7ff64405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:10.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.227+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7ff6480766d0 con 0x7ff6540f65d0 2021-11-20T07:43:10.236 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.227+0000 7ff653fff700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 0x7ff64405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:10.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.227+0000 7ff653fff700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 0x7ff64405e7c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff640004e30 tx=0x7ff640009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:10.237 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.230+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff648048e60 con 0x7ff6540f65d0 2021-11-20T07:43:10.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:09 smithi112 conmon[36062]: audit 2021-11-20T07:43:09.083664+0000 mon.smithi071 (mon.0) 909 : audit [DBG] from='client.? 172.21.15.71:0/1604589260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:10.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:09 smithi112 conmon[36062]: 2021-11-20T07:43:10.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.411+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6540fee80 con 0x7ff6540f65d0 2021-11-20T07:43:10.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.412+0000 7ff6517fa700 1 -- 172.21.15.71:0/4198452086 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7ff6540fee80 con 0x7ff6540f65d0 2021-11-20T07:43:10.414 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:10.415 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:10.416 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.413+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 msgr2=0x7ff64405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:10.416 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.413+0000 7ff659fc0700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 0x7ff64405e7c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff640004e30 tx=0x7ff640009b30).stop 2021-11-20T07:43:10.417 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 msgr2=0x7ff654106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:10.417 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ff648002700 tx=0x7ff648009e00).stop 2021-11-20T07:43:10.417 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 shutdown_connections 2021-11-20T07:43:10.417 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7ff64405c300 0x7ff64405e7c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6540f65d0 0x7ff654106d30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 --2- 172.21.15.71:0/4198452086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6540f7870 0x7ff654107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:10.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 >> 172.21.15.71:0/4198452086 conn(0x7ff6540f1d10 msgr2=0x7ff6540fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:10.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 shutdown_connections 2021-11-20T07:43:10.418 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:10.414+0000 7ff659fc0700 1 -- 172.21.15.71:0/4198452086 wait complete. 2021-11-20T07:43:10.419 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:10.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:10.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:11.283 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:10 smithi112 conmon[36062]: audit 2021-11-20T07:43:10.413448+0000 mon.smithi071 (mon.0) 910 : audit [DBG] from='client.? 172.21.15.71:0/4198452086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:11.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:10 smithi112 conmon[36062]: cluster 2021-11-20T07:43:10.430871+0000 mgr.smithi071.ynqmwn (mgr.14162) 336 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:11.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:10 smithi071 conmon[33687]: audit 2021-11-20T07:43:10.413448+0000 mon.smithi071 (mon.0) 910 : audit [DBG] from='client.? 172.21.15.71:0/4198452086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:11.318 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:10 smithi071 conmon[33687]: cluster 2021-11-20T07 2021-11-20T07:43:11.319 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:10 smithi071 conmon[33687]: :43:10.430871+0000 mgr.smithi071.ynqmwn (mgr.14162) 336 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T07:43:11.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:11.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- 172.21.15.71:0/2526603160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 msgr2=0x7f29840f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:11.559 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 --2- 172.21.15.71:0/2526603160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f29840f7cd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f2978004660 tx=0x7f2978009b30).stop 2021-11-20T07:43:11.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- 172.21.15.71:0/2526603160 shutdown_connections 2021-11-20T07:43:11.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 --2- 172.21.15.71:0/2526603160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f29840f7cd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:11.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 --2- 172.21.15.71:0/2526603160 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 0x7f29840f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:11.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- 172.21.15.71:0/2526603160 >> 172.21.15.71:0/2526603160 conn(0x7f29840f1d10 msgr2=0x7f29840f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:11.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- 172.21.15.71:0/2526603160 shutdown_connections 2021-11-20T07:43:11.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- 172.21.15.71:0/2526603160 wait complete. 2021-11-20T07:43:11.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 Processor -- start 2021-11-20T07:43:11.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 -- start start 2021-11-20T07:43:11.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.553+0000 7f298a08f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 0x7f298406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:11.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298a08f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:11.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298a08f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298406a410 con 0x7f29840f7870 2021-11-20T07:43:11.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298a08f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298406a550 con 0x7f29840f65d0 2021-11-20T07:43:11.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:11.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49840/0 (socket says 172.21.15.71:49840) 2021-11-20T07:43:11.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 -- 172.21.15.71:0/1566841370 learned_addr learned my addr 172.21.15.71:0/1566841370 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:11.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298908d700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 0x7f298406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:11.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 -- 172.21.15.71:0/1566841370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 msgr2=0x7f298406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:11.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 0x7f298406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:11.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 -- 172.21.15.71:0/1566841370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978005040 con 0x7f29840f7870 2021-11-20T07:43:11.565 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298888c700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f2978009830 tx=0x7f2978009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:11.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2978018070 con 0x7f29840f7870 2021-11-20T07:43:11.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2978013410 con 0x7f29840f7870 2021-11-20T07:43:11.566 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298406a7d0 con 0x7f29840f7870 2021-11-20T07:43:11.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297801c620 con 0x7f29840f7870 2021-11-20T07:43:11.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.554+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298410ed60 con 0x7f29840f7870 2021-11-20T07:43:11.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.555+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2984045ad0 con 0x7f29840f7870 2021-11-20T07:43:11.567 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.556+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f297801c8e0 con 0x7f29840f7870 2021-11-20T07:43:11.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.556+0000 7f2981ffb700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 0x7f296c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:11.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.556+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f2978078350 con 0x7f29840f7870 2021-11-20T07:43:11.568 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.556+0000 7f298908d700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 0x7f296c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:11.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.556+0000 7f298908d700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 0x7f296c05e7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f298406b730 tx=0x7f2974008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:11.569 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.560+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f297804aa00 con 0x7f29840f7870 2021-11-20T07:43:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.743+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2984027a00 con 0x7f29840f7870 2021-11-20T07:43:11.746 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.744+0000 7f2981ffb700 1 -- 172.21.15.71:0/1566841370 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f297804e020 con 0x7f29840f7870 2021-11-20T07:43:11.747 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:11.747 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 msgr2=0x7f296c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 0x7f296c05e7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f298406b730 tx=0x7f2974008040).stop 2021-11-20T07:43:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 msgr2=0x7f2984069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f2978009830 tx=0x7f2978009b30).stop 2021-11-20T07:43:11.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 shutdown_connections 2021-11-20T07:43:11.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.745+0000 7f298a08f700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29840f65d0 0x7f298406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:11.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.746+0000 7f298a08f700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f296c05c300 0x7f296c05e7c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:11.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.746+0000 7f298a08f700 1 --2- 172.21.15.71:0/1566841370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29840f7870 0x7f2984069e20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:11.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.746+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 >> 172.21.15.71:0/1566841370 conn(0x7f29840f1d10 msgr2=0x7f29840fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:11.751 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.746+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 shutdown_connections 2021-11-20T07:43:11.752 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:11.746+0000 7f298a08f700 1 -- 172.21.15.71:0/1566841370 wait complete. 2021-11-20T07:43:11.752 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:11.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:11.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:12.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:12 smithi112 conmon[36062]: audit 2021-11-20T07:43:11.745308+0000 mon.smithi071 (mon.0) 911 2021-11-20T07:43:12.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:12 smithi112 conmon[36062]: : audit [DBG] from='client.? 172.21.15.71:0/1566841370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:12.788 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:12 smithi071 conmon[33687]: audit 2021-11-20T07:43:11.745308+0000 mon.smithi071 (mon.0) 911 : audit [DBG] from='client.? 172.21.15.71:0/1566841370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:12.889 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 -- 172.21.15.71:0/1117014791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f6820 msgr2=0x7f5d200f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:12.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 --2- 172.21.15.71:0/1117014791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f6820 0x7f5d200f6c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f5d10004660 tx=0x7f5d10009b30).stop 2021-11-20T07:43:12.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 -- 172.21.15.71:0/1117014791 shutdown_connections 2021-11-20T07:43:12.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 --2- 172.21.15.71:0/1117014791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f6820 0x7f5d200f6c80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:12.890 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 --2- 172.21.15.71:0/1117014791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f5580 0x7f5d200f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:12.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 -- 172.21.15.71:0/1117014791 >> 172.21.15.71:0/1117014791 conn(0x7f5d200f0ce0 msgr2=0x7f5d200f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:12.891 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 -- 172.21.15.71:0/1117014791 shutdown_connections 2021-11-20T07:43:12.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.885+0000 7f5d279bc700 1 -- 172.21.15.71:0/1117014791 wait complete. 2021-11-20T07:43:12.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 Processor -- start 2021-11-20T07:43:12.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 -- start start 2021-11-20T07:43:12.892 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:12.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f6820 0x7f5d2010bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:12.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2010c120 con 0x7f5d200f5580 2021-11-20T07:43:12.894 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2010c260 con 0x7f5d200f6820 2021-11-20T07:43:12.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:12.895 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49848/0 (socket says 172.21.15.71:49848) 2021-11-20T07:43:12.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 -- 172.21.15.71:0/3208597156 learned_addr learned my addr 172.21.15.71:0/3208597156 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:12.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d24f57700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f6820 0x7f5d2010bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:12.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 -- 172.21.15.71:0/3208597156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f6820 msgr2=0x7f5d2010bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:12.896 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f6820 0x7f5d2010bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:12.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 -- 172.21.15.71:0/3208597156 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10005040 con 0x7f5d200f5580 2021-11-20T07:43:12.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d25758700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f5d1c002700 tx=0x7f5d1c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:12.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c0079c0 con 0x7f5d200f5580 2021-11-20T07:43:12.897 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d1c00f040 con 0x7f5d200f5580 2021-11-20T07:43:12.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2010c4c0 con 0x7f5d200f5580 2021-11-20T07:43:12.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.886+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c014850 con 0x7f5d200f5580 2021-11-20T07:43:12.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.887+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2010ca50 con 0x7f5d200f5580 2021-11-20T07:43:12.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.888+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d20027a00 con 0x7f5d200f5580 2021-11-20T07:43:12.898 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.888+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5d1c01b060 con 0x7f5d200f5580 2021-11-20T07:43:12.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.888+0000 7f5d15ffb700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 0x7f5d0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:12.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.888+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f5d1c076390 con 0x7f5d200f5580 2021-11-20T07:43:12.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.888+0000 7f5d24f57700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 0x7f5d0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:12.900 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.889+0000 7f5d24f57700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 0x7f5d0c05e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f5d10009f50 tx=0x7f5d10009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:12.901 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:12.892+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d1c048aa0 con 0x7f5d200f5580 2021-11-20T07:43:13.077 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.075+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d200fdd10 con 0x7f5d200f5580 2021-11-20T07:43:13.078 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.076+0000 7f5d15ffb700 1 -- 172.21.15.71:0/3208597156 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f5d1c04c0c0 con 0x7f5d200f5580 2021-11-20T07:43:13.078 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:13.078 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 msgr2=0x7f5d0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 0x7f5d0c05e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f5d10009f50 tx=0x7f5d10009b30).stop 2021-11-20T07:43:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 msgr2=0x7f5d20060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f5d1c002700 tx=0x7f5d1c009e00).stop 2021-11-20T07:43:13.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 shutdown_connections 2021-11-20T07:43:13.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f5d0c05c2a0 0x7f5d0c05e760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:13.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d200f5580 0x7f5d20060550 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:13.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 --2- 172.21.15.71:0/3208597156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5d200f6820 0x7f5d2010bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:13.081 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 >> 172.21.15.71:0/3208597156 conn(0x7f5d200f0ce0 msgr2=0x7f5d200fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:13.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 shutdown_connections 2021-11-20T07:43:13.082 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:13.078+0000 7f5d279bc700 1 -- 172.21.15.71:0/3208597156 wait complete. 2021-11-20T07:43:13.083 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:13.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:13.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:13.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:13 smithi112 conmon[36062]: cluster 2021-11-20T07:43:12.431252+0000 mgr.smithi071.ynqmwn (mgr.14162) 337 : cluster [DBG] 2021-11-20T07:43:13.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:13 smithi112 conmon[36062]: pgmap v315: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:43:13.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:13 smithi112 conmon[36062]: audit 2021-11-20T07:43:13.077386+0000 mon.smithi071 (mon.0) 912 : audit [DBG] from='client.? 172.21.15.71:0/3208597156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:13 smithi071 conmon[33687]: cluster 2021-11-20T07:43:12.431252+0000 mgr.smithi071.ynqmwn (mgr.14162) 337 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T07:43:13.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:13 smithi071 conmon[33687]: audit 2021-11-20T 2021-11-20T07:43:13.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:13 smithi071 conmon[33687]: 07:43:13.077386+0000 mon.smithi071 (mon.0) 912 : audit [DBG] from='client.? 172.21.15.71:0/3208597156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:14.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:14.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 -- 172.21.15.71:0/3915082640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f55a0 msgr2=0x7f2c140f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:14.221 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 --2- 172.21.15.71:0/3915082640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f55a0 0x7f2c140f59c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f2c08004660 tx=0x7f2c08009b30).stop 2021-11-20T07:43:14.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 -- 172.21.15.71:0/3915082640 shutdown_connections 2021-11-20T07:43:14.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 --2- 172.21.15.71:0/3915082640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f6840 0x7f2c140f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 --2- 172.21.15.71:0/3915082640 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f55a0 0x7f2c140f59c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.222 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 -- 172.21.15.71:0/3915082640 >> 172.21.15.71:0/3915082640 conn(0x7f2c140f0ce0 msgr2=0x7f2c140f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:14.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 -- 172.21.15.71:0/3915082640 shutdown_connections 2021-11-20T07:43:14.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.216+0000 7f2c19704700 1 -- 172.21.15.71:0/3915082640 wait complete. 2021-11-20T07:43:14.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 Processor -- start 2021-11-20T07:43:14.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 -- start start 2021-11-20T07:43:14.223 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f55a0 0x7f2c14105d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:14.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:14.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c141067e0 con 0x7f2c140f6840 2021-11-20T07:43:14.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c19704700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c14106920 con 0x7f2c140f55a0 2021-11-20T07:43:14.224 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c127fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:14.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c127fc700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49860/0 (socket says 172.21.15.71:49860) 2021-11-20T07:43:14.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.217+0000 7f2c127fc700 1 -- 172.21.15.71:0/179850370 learned_addr learned my addr 172.21.15.71:0/179850370 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:14.225 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c127fc700 1 -- 172.21.15.71:0/179850370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f55a0 msgr2=0x7f2c14105d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:14.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c12ffd700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f55a0 0x7f2c14105d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:14.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c127fc700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f55a0 0x7f2c14105d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c127fc700 1 -- 172.21.15.71:0/179850370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08005040 con 0x7f2c140f6840 2021-11-20T07:43:14.226 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c127fc700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f2c0400e7f0 tx=0x7f2c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:14.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c040092e0 con 0x7f2c140f6840 2021-11-20T07:43:14.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c0400f040 con 0x7f2c140f6840 2021-11-20T07:43:14.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1410b3c0 con 0x7f2c140f6840 2021-11-20T07:43:14.227 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c040146c0 con 0x7f2c140f6840 2021-11-20T07:43:14.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.218+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1410b980 con 0x7f2c140f6840 2021-11-20T07:43:14.228 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.219+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c14046330 con 0x7f2c140f6840 2021-11-20T07:43:14.229 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.219+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2c0401b030 con 0x7f2c140f6840 2021-11-20T07:43:14.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.219+0000 7f2bfb7fe700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 0x7f2bfc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:14.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.220+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f2c04076140 con 0x7f2c140f6840 2021-11-20T07:43:14.230 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.220+0000 7f2c12ffd700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 0x7f2bfc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:14.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.220+0000 7f2c12ffd700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 0x7f2bfc05e760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f2c080043d0 tx=0x7f2c0800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:14.231 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.223+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c040488a0 con 0x7f2c140f6840 2021-11-20T07:43:14.408 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.406+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c14027a00 con 0x7f2c140f6840 2021-11-20T07:43:14.409 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.407+0000 7f2bfb7fe700 1 -- 172.21.15.71:0/179850370 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f2c04019090 con 0x7f2c140f6840 2021-11-20T07:43:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:14.409 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:14.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 msgr2=0x7f2bfc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:14.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 0x7f2bfc05e760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f2c080043d0 tx=0x7f2c0800b040).stop 2021-11-20T07:43:14.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 msgr2=0x7f2c14106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:14.411 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f2c0400e7f0 tx=0x7f2c0400b040).stop 2021-11-20T07:43:14.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 shutdown_connections 2021-11-20T07:43:14.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c140f55a0 0x7f2c14105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.412 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f2bfc05c2a0 0x7f2bfc05e760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 --2- 172.21.15.71:0/179850370 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c140f6840 0x7f2c14106230 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:14.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 >> 172.21.15.71:0/179850370 conn(0x7f2c140f0ce0 msgr2=0x7f2c140fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:14.413 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 shutdown_connections 2021-11-20T07:43:14.414 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:14.408+0000 7f2c19704700 1 -- 172.21.15.71:0/179850370 wait complete. 2021-11-20T07:43:14.414 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:14.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:14.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:14 smithi112 conmon[36062]: audit 2021-11-20T07:43:14.408229+0000 mon.smithi071 (mon.0) 913 : audit [DBG] 2021-11-20T07:43:14.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:14 smithi112 conmon[36062]: from='client.? 172.21.15.71:0/179850370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:14.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:14 smithi071 conmon[33687]: audit 2021-11-20T07:43:14.408229+0000 mon.smithi071 (mon.0) 913 : audit [DBG] from='client.? 172.21.15.71:0/179850370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:15.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:15.552 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 -- 172.21.15.71:0/133535736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 msgr2=0x7f47e00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:15.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/133535736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e00f59a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f47dc004660 tx=0x7f47dc009b30).stop 2021-11-20T07:43:15.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 -- 172.21.15.71:0/133535736 shutdown_connections 2021-11-20T07:43:15.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/133535736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 0x7f47e00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.553 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/133535736 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e00f59a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 -- 172.21.15.71:0/133535736 >> 172.21.15.71:0/133535736 conn(0x7f47e00f0ce0 msgr2=0x7f47e00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:15.554 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 -- 172.21.15.71:0/133535736 shutdown_connections 2021-11-20T07:43:15.555 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.547+0000 7f47e7b7f700 1 -- 172.21.15.71:0/133535736 wait complete. 2021-11-20T07:43:15.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 Processor -- start 2021-11-20T07:43:15.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 -- start start 2021-11-20T07:43:15.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:15.556 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 0x7f47e010bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:15.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e010c190 con 0x7f47e00f5580 2021-11-20T07:43:15.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e7b7f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e010c2d0 con 0x7f47e00f6820 2021-11-20T07:43:15.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:15.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49868/0 (socket says 172.21.15.71:49868) 2021-11-20T07:43:15.557 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 -- 172.21.15.71:0/529046326 learned_addr learned my addr 172.21.15.71:0/529046326 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:15.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e511a700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 0x7f47e010bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:15.558 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 -- 172.21.15.71:0/529046326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 msgr2=0x7f47e010bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:15.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 0x7f47e010bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.548+0000 7f47e591b700 1 -- 172.21.15.71:0/529046326 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc005040 con 0x7f47e00f5580 2021-11-20T07:43:15.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47e591b700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f47dc009f50 tx=0x7f47dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:15.560 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47dc018070 con 0x7f47e00f5580 2021-11-20T07:43:15.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47dc013410 con 0x7f47e00f5580 2021-11-20T07:43:15.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47dc01de70 con 0x7f47e00f5580 2021-11-20T07:43:15.561 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e010c4d0 con 0x7f47e00f5580 2021-11-20T07:43:15.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.549+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e010ca30 con 0x7f47e00f5580 2021-11-20T07:43:15.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.550+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e0027a00 con 0x7f47e00f5580 2021-11-20T07:43:15.562 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.550+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f47dc01d420 con 0x7f47e00f5580 2021-11-20T07:43:15.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.550+0000 7f47d67fc700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 0x7f47cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:15.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.550+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f47dc078020 con 0x7f47e00f5580 2021-11-20T07:43:15.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.550+0000 7f47e511a700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 0x7f47cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:15.563 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.551+0000 7f47e511a700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 0x7f47cc05e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f47d00096c0 tx=0x7f47d0009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:15.564 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.554+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47dc04a7b0 con 0x7f47e00f5580 2021-11-20T07:43:15.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.740+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47e00fdd10 con 0x7f47e00f5580 2021-11-20T07:43:15.745 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.740+0000 7f47d67fc700 1 -- 172.21.15.71:0/529046326 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f47dc04ddd0 con 0x7f47e00f5580 2021-11-20T07:43:15.746 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:15.746 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:01.425880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24529":{"gid":24529,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":80,"state":"up:active","state_seq":14,"addr":"172.21.15.112:6825/1889679177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":1889679177},{"type":"v1","addr":"172.21.15.112:6825","nonce":1889679177}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:15.747 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 msgr2=0x7f47cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:15.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 0x7f47cc05e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f47d00096c0 tx=0x7f47d0009be0).stop 2021-11-20T07:43:15.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 msgr2=0x7f47e0060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:15.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f47dc009f50 tx=0x7f47dc009b30).stop 2021-11-20T07:43:15.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 shutdown_connections 2021-11-20T07:43:15.748 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f47cc05c2a0 0x7f47cc05e760 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47e00f5580 0x7f47e0060550 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 --2- 172.21.15.71:0/529046326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47e00f6820 0x7f47e010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:15.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 >> 172.21.15.71:0/529046326 conn(0x7f47e00f0ce0 msgr2=0x7f47e00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:15.749 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 shutdown_connections 2021-11-20T07:43:15.750 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:15.742+0000 7f47e7b7f700 1 -- 172.21.15.71:0/529046326 wait complete. 2021-11-20T07:43:15.750 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2021-11-20T07:43:15.754 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:15 smithi071 conmon[33687]: cluster 2021-11-20T07:43:14.431592+0000 mgr.smithi071.ynqmwn (mgr.14162) 338 : cluster 2021-11-20T07:43:15.755 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:15 smithi071 conmon[33687]: [DBG] pgmap v316: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:43:15.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:01.425880+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24529': {'gid': 24529, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.112:6825/1889679177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 1889679177}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 1889679177}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:15.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:15.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:15 smithi112 conmon[36062]: cluster 2021-11-20T07:43:14.431592+0000 mgr.smithi071.ynqmwn (mgr.14162) 338 : cluster [DBG] 2021-11-20T07:43:15.783 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:15 smithi112 conmon[36062]: pgmap v316: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T07:43:16.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T07:43:16.783 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: audit 2021-11-20T07:43:15.741889+0000 mon.smithi071 (mon.0) 914 : audit 2021-11-20T07:43:16.784 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: [DBG] from='client.? 172.21.15.71:0/529046326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:16.784 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: cluster 2021-11-20T07:43:16.308490+0000 mon.smithi071 (mon.0) 915 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:43:16.784 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: cluster 2021-11-20T07:43:16.784 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: 2021-11-20T07:43:16.312310+0000 mon.smithi071 (mon.0) 916 : cluster 2021-11-20T07:43:16.784 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T07:43:16.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:43:16.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: 20T07:43:16.431949+0000 2021-11-20T07:43:16.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: mgr.smithi071.ynqmwn (mgr.14162) 339 : cluster [DBG] 2021-11-20T07:43:16.785 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:16 smithi071 conmon[33687]: pgmap v318: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:43:16.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:16 smithi112 conmon[36062]: audit 2021-11-20T07:43:15.741889+0000 mon.smithi071 (mon.0) 914 : audit [DBG] from='client.? 172.21.15.71:0/529046326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:16.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:16 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.308490+0000 mon.smithi071 (mon.0) 915 : cluster [INF] daemon mds.cephfs.smithi112.bxcrgi restarted 2021-11-20T07:43:16.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:16 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.312310+0000 mon.smithi071 (mon.0) 916 : cluster [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T07:43:16.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:16 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.431949+0000 mgr.smithi071.ynqmwn (mgr.14162) 2021-11-20T07:43:16.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:16 smithi112 conmon[36062]: 339 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2021-11-20T07:43:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 -- 172.21.15.71:0/2478821516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 msgr2=0x7f53840fb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 --2- 172.21.15.71:0/2478821516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f53840fb5f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f5374004660 tx=0x7f5374009b30).stop 2021-11-20T07:43:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 -- 172.21.15.71:0/2478821516 shutdown_connections 2021-11-20T07:43:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 --2- 172.21.15.71:0/2478821516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53840fbae0 0x7f53840fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 --2- 172.21.15.71:0/2478821516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f53840fb5f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 -- 172.21.15.71:0/2478821516 >> 172.21.15.71:0/2478821516 conn(0x7f53840f0c10 msgr2=0x7f53840f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 -- 172.21.15.71:0/2478821516 shutdown_connections 2021-11-20T07:43:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.875+0000 7f538939b700 1 -- 172.21.15.71:0/2478821516 wait complete. 2021-11-20T07:43:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 Processor -- start 2021-11-20T07:43:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 -- start start 2021-11-20T07:43:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53840fbae0 0x7f53841094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384109af0 con 0x7f5384060410 2021-11-20T07:43:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f538939b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538410a3a0 con 0x7f53840fbae0 2021-11-20T07:43:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49878/0 (socket says 172.21.15.71:49878) 2021-11-20T07:43:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 -- 172.21.15.71:0/3842906639 learned_addr learned my addr 172.21.15.71:0/3842906639 (peer_addr_for_me v2:172.21.15.71:0/0) 2021-11-20T07:43:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 -- 172.21.15.71:0/3842906639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53840fbae0 msgr2=0x7f53841094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53840fbae0 0x7f53841094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 -- 172.21.15.71:0/3842906639 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374005040 con 0x7f5384060410 2021-11-20T07:43:16.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.876+0000 7f5382ffd700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f5374009830 tx=0x7f5374009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:16.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5374018070 con 0x7f5384060410 2021-11-20T07:43:16.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5374013410 con 0x7f5384060410 2021-11-20T07:43:16.884 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538410a620 con 0x7f5384060410 2021-11-20T07:43:16.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537401c620 con 0x7f5384060410 2021-11-20T07:43:16.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538410ab80 con 0x7f5384060410 2021-11-20T07:43:16.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.877+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53840f48a0 con 0x7f5384060410 2021-11-20T07:43:16.885 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.878+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f537401c890 con 0x7f5384060410 2021-11-20T07:43:16.886 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.878+0000 7f536b7fe700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 0x7f536c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T07:43:16.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.878+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6930+0+0 (secure 0 0 0) 0x7f53740782c0 con 0x7f5384060410 2021-11-20T07:43:16.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.878+0000 7f53827fc700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 0x7f536c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T07:43:16.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.879+0000 7f53827fc700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 0x7f536c05e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f53840601e0 tx=0x7f5378006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T07:43:16.887 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:16.882+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f537404a990 con 0x7f5384060410 2021-11-20T07:43:17.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.067+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5384045ad0 con 0x7f5384060410 2021-11-20T07:43:17.069 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.067+0000 7f536b7fe700 1 -- 172.21.15.71:0/3842906639 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2201 (secure 0 0 0) 0x7f537404dfb0 con 0x7f5384060410 2021-11-20T07:43:17.070 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-11-20T07:43:17.070 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T07:38:12.408568+0000","modified":"2021-11-20T07:43:16.448499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24631,"mds_1":24651},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24631":{"gid":24631,"name":"cephfs.smithi112.iqrmps","rank":0,"incarnation":85,"state":"up:resolve","state_seq":2,"addr":"172.21.15.112:6827/503879816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":503879816},{"type":"v1","addr":"172.21.15.112:6827","nonce":503879816}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24651":{"gid":24651,"name":"cephfs.smithi112.bxcrgi","rank":1,"incarnation":88,"state":"up:replay","state_seq":2,"addr":"172.21.15.112:6825/2286023960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2286023960},{"type":"v1","addr":"172.21.15.112:6825","nonce":2286023960}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T07:43:17.071 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 msgr2=0x7f536c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:17.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 0x7f536c05e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f53840601e0 tx=0x7f5378006ab0).stop 2021-11-20T07:43:17.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 msgr2=0x7f5384108fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T07:43:17.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f5374009830 tx=0x7f5374009b30).stop 2021-11-20T07:43:17.072 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 shutdown_connections 2021-11-20T07:43:17.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:6800/429188327,v1:172.21.15.71:6801/429188327] conn(0x7f536c05c300 0x7f536c05e7c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:17.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5384060410 0x7f5384108fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T07:43:17.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 --2- 172.21.15.71:0/3842906639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53840fbae0 0x7f53841094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T07:43:17.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 >> 172.21.15.71:0/3842906639 conn(0x7f53840f0c10 msgr2=0x7f53840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T07:43:17.073 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 shutdown_connections 2021-11-20T07:43:17.074 INFO:teuthology.orchestra.run.smithi071.stderr:2021-11-20T07:43:17.069+0000 7f538939b700 1 -- 172.21.15.71:0/3842906639 wait complete. 2021-11-20T07:43:17.074 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 88 2021-11-20T07:43:17.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T07:38:12.408568+0000', 'modified': '2021-11-20T07:43:16.448499+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24631, 'mds_1': 24651}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24631': {'gid': 24631, 'name': 'cephfs.smithi112.iqrmps', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.112:6827/503879816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6826', 'nonce': 503879816}, {'type': 'v1', 'addr': '172.21.15.112:6827', 'nonce': 503879816}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24651': {'gid': 24651, 'name': 'cephfs.smithi112.bxcrgi', 'rank': 1, 'incarnation': 88, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.112:6825/2286023960', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6824', 'nonce': 2286023960}, {'type': 'v1', 'addr': '172.21.15.112:6825', 'nonce': 2286023960}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T07:43:17.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24631:up:resolve 2021-11-20T07:43:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.440936+0000 mon.smithi071 (mon.0) 917 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T07:43:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.448398+0000 mon.smithi071 (mon.0) 918 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2286023960,v1:172.21.15.112:6825/2286023960] up:boot 2021-11-20T07:43:17.784 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.448461+0000 mon.smithi071 (mon.0) 919 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:43:17.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:17.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: 2021-11-20T07:43:16.448561+0000 mon.smithi071 (mon.0) 920 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:43:17.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16 2021-11-20T07:43:17.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: .448755+0000 mon.smithi071 (mon.0) 921 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi112.iqrmps=up:resolve} 1 up:standby, 1 failed 2021-11-20T07:43:17.785 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: audit 2021-11 2021-11-20T07:43:17.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: -20T07:43:16.448857+0000 mon.smithi071 (mon.0) 922 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:43:17.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: cluster 2021-11-20T07:43:16.451310+0000 2021-11-20T07:43:17.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: mon.smithi071 (mon.0) 923 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2021-11-20T07:43:17.786 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: audit 2021-11-20T07:43:17.068897+0000 mon.smithi071 ( 2021-11-20T07:43:17.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:17 smithi112 conmon[36062]: mon.0) 924 : audit [DBG] from='client.? 172.21.15.71:0/3842906639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11-20T07:43:16.440936+0000 mon.smithi071 (mon.0) 917 : cluster [INF] 2021-11-20T07:43:17.818 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T07:43:17.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11-20T07:43:16.448398+0000 mon.smithi071 (mon.0) 918 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2286023960,v1:172.21.15.112:6825/2286023960] up:boot 2021-11-20T07:43:17.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11-20T07:43:17.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: 2021-11-20T07:43:16.448461+0000 mon.smithi071 (mon.0) 919 : cluster [INF] Standby daemon mds.cephfs.smithi112.bxcrgi assigned to filesystem cephfs as rank 1 2021-11-20T07:43:17.819 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11-20T07: 2021-11-20T07:43:17.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: 43:16.448561+0000 mon.smithi071 (mon.0 2021-11-20T07:43:17.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: ) 920 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T07:43:17.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11 2021-11-20T07:43:17.820 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: -20T07:43:16.448755+0000 mon.smithi071 (mon 2021-11-20T07:43:17.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: .0) 921 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi112.iqrmps=up:resolve} 1 up:standby, 1 failed 2021-11-20T07:43:17.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:43:17.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: -20T07:43:16.448857+0000 mon.smithi071 ( 2021-11-20T07:43:17.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: mon.0) 922 : audit [DBG] from='mgr.14162 172.21.15.71:0/291309789' entity='mgr.smithi071.ynqmwn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.bxcrgi"}]: dispatch 2021-11-20T07:43:17.821 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: cluster 2021-11- 2021-11-20T07:43:17.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: 20T07:43:16.451310+0000 mon.smithi071 (mon 2021-11-20T07:43:17.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: .0) 923 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:replay} 2021-11-20T07:43:17.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: audit 2021-11 2021-11-20T07:43:17.822 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: -20T07:43:17.068897+0000 mon.smithi071 2021-11-20T07:43:17.823 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:17 smithi071 conmon[33687]: (mon.0) 924 : audit [DBG] from='client.? 172.21.15.71:0/3842906639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T07:43:18.085 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-11-20T07:38:12.408568+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_24631": { "addr": "172.21.15.112:6827/503879816", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6826", "nonce": 503879816, "type": "v2" }, { "addr": "172.21.15.112:6827", "nonce": 503879816, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24631, "incarnation": 85, "join_fscid": 1, "name": "cephfs.smithi112.iqrmps", "rank": 0, "state": "up:resolve", "state_seq": 2 }, "gid_24651": { "addr": "172.21.15.112:6825/2286023960", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6824", "nonce": 2286023960, "type": "v2" }, { "addr": "172.21.15.112:6825", "nonce": 2286023960, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24651, "incarnation": 88, "join_fscid": -1, "name": "cephfs.smithi112.bxcrgi", "rank": 1, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 2, "modified": "2021-11-20T07:43:16.448499+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 1, "stopped": [], "tableserver": 0, "up": { "mds_0": 24631, "mds_1": 24651 } } } ], "standbys": [] } 2021-11-20T07:43:18.086 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T07:43:18.214 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1982f3c7b22e4a23a9bc906e34fbd542 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T07:43:18.217 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2021-11-20T07:43:18.230 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-11-20T07:43:18.277 INFO:tasks.cephadm:Teardown begin 2021-11-20T07:43:18.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:43:18.313 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:43:18.360 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-11-20T07:43:18.361 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T07:43:18.382 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T07:43:18.403 INFO:tasks.cephadm:Stopping all daemons... 2021-11-20T07:43:18.404 INFO:tasks.cephadm.mon.smithi071:Stopping mon.smithi071... 2021-11-20T07:43:18.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071 2021-11-20T07:43:18.568 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 systemd[1]: Stopping Ceph mon.smithi071 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:18.815 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 conmon[33687]: cluster 2021-11-20T07:43:17.452314+0000 mon.smithi071 (mon.0) 925 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2286023960,v1:172.21.15.112:6825/2286023960] up:resolve 2021-11-20T07:43:18.815 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 conmon[33687]: cluster 2021-11-20T07:43:17.452361+0000 mon.smithi071 (mon.0) 926 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2021-11-20T07:43:18.816 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 conmon[33687]: cluster 2021-11-20T07:43:18.432445+0000 mgr.smithi071.ynqmwn (mgr.14162) 340 : cluster [DBG] pgmap v319: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T07:43:18.817 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:18 smithi112 conmon[36062]: cluster 2021-11-20T07:43:17.452314+0000 mon.smithi071 (mon.0) 925 : cluster [DBG] mds.? [v2:172.21.15.112:6824/2286023960,v1:172.21.15.112:6825/2286023960] up:resolve 2021-11-20T07:43:18.818 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:18 smithi112 conmon[36062]: cluster 2021-11-20T07:43:17.452361+0000 mon.smithi071 (mon.0) 926 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi112.iqrmps=up:resolve,1=cephfs.smithi112.bxcrgi=up:resolve} 2021-11-20T07:43:18.819 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:18 smithi112 conmon[36062]: cluster 2021-11-20T07:43:18.432445+0000 mgr.smithi071.ynqmwn (mgr.14162) 340 : cluster [DBG] pgmap v319: 65 pgs: 65 active+clean; 5.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T07:43:18.872 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 conmon[33687]: debug 2021-11-20T07:43:18.573+0000 7f9c1d36f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi071 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T07:43:18.872 INFO:journalctl@ceph.mon.smithi071.smithi071.stdout:Nov 20 07:43:18 smithi071 conmon[33687]: debug 2021-11-20T07:43:18.573+0000 7f9c1d36f700 -1 mon.smithi071@0(leader) e2 *** Got Signal Terminated *** 2021-11-20T07:43:18.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi071.service' 2021-11-20T07:43:19.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:19.015 INFO:tasks.cephadm.mon.smithi071:Stopped mon.smithi071 2021-11-20T07:43:19.016 INFO:tasks.cephadm.mon.smithi112:Stopping mon.smithi112... 2021-11-20T07:43:19.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi112 2021-11-20T07:43:19.535 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:19 smithi112 systemd[1]: Stopping Ceph mon.smithi112 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:19.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:19 smithi112 conmon[36062]: debug 2021-11-20T07:43:19.182+0000 7f147ab59700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi112 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T07:43:19.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:19 smithi112 conmon[36062]: debug 2021-11-20T07:43:19.182+0000 7f147ab59700 -1 mon.smithi112@1(peon) e2 *** Got Signal Terminated *** 2021-11-20T07:43:19.979 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 20 07:43:19 smithi112 podman[59211]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-mon.smithi112 2021-11-20T07:43:19.982 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@mon.smithi112.service' 2021-11-20T07:43:20.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:20.038 INFO:tasks.cephadm.mon.smithi112:Stopped mon.smithi112 2021-11-20T07:43:20.038 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-11-20T07:43:20.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.0 2021-11-20T07:43:20.287 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 systemd[1]: Stopping Ceph osd.0 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:20.287 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 conmon[50727]: debug 2021-11-20T07:43:20.159+0000 7fac3ed6e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:20.288 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 conmon[50727]: debug 2021-11-20T07:43:20.159+0000 7fac3ed6e700 -1 osd.0 59 *** Got signal Terminated *** 2021-11-20T07:43:20.288 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 conmon[50727]: debug 2021-11-20T07:43:20.159+0000 7fac3ed6e700 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:20.568 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 podman[96215]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.0 2021-11-20T07:43:20.947 INFO:journalctl@ceph.osd.0.smithi071.stdout:Nov 20 07:43:20 smithi071 podman[96389]: 2021-11-20T07:43:21.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.0.service' 2021-11-20T07:43:21.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:21.567 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-11-20T07:43:21.567 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-11-20T07:43:21.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.1 2021-11-20T07:43:21.776 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:21 smithi071 systemd[1]: Stopping Ceph osd.1 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:22.068 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:21 smithi071 conmon[55662]: debug 2021-11-20T07:43:21.772+0000 7f03d2335700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:22.068 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:21 smithi071 conmon[55662]: debug 2021-11-20T07:43:21.772+0000 7f03d2335700 -1 osd.1 59 *** Got signal Terminated *** 2021-11-20T07:43:22.069 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:21 smithi071 conmon[55662]: debug 2021-11-20T07:43:21.772+0000 7f03d2335700 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:22.069 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:22 smithi071 podman[96547]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.1 2021-11-20T07:43:22.568 INFO:journalctl@ceph.osd.1.smithi071.stdout:Nov 20 07:43:22 smithi071 podman[96710]: 2021-11-20T07:43:22.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.1.service' 2021-11-20T07:43:23.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:23.028 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-11-20T07:43:23.028 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-11-20T07:43:23.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.2 2021-11-20T07:43:23.250 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 systemd[1]: Stopping Ceph osd.2 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:23.251 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 conmon[62032]: debug 2021-11-20T07:43:23.183+0000 7f1443750700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:23.252 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 conmon[62032]: debug 2021-11-20T07:43:23.183+0000 7f1443750700 -1 osd.2 59 *** Got signal Terminated *** 2021-11-20T07:43:23.253 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 conmon[62032]: debug 2021-11-20T07:43:23.183+0000 7f1443750700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:23.818 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 podman[96866]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.2 2021-11-20T07:43:24.318 INFO:journalctl@ceph.osd.2.smithi071.stdout:Nov 20 07:43:23 smithi071 podman[97036]: 2021-11-20T07:43:24.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.2.service' 2021-11-20T07:43:24.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:24.759 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-11-20T07:43:24.759 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-11-20T07:43:24.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.3 2021-11-20T07:43:25.198 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:24 smithi112 systemd[1]: Stopping Ceph osd.3 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:25.204 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:24 smithi112 conmon[42410]: debug 2021-11-20T07:43:24.893+0000 7fc83027d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:25.205 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:24 smithi112 conmon[42410]: debug 2021-11-20T07:43:24.893+0000 7fc83027d700 -1 osd.3 59 *** Got signal Terminated *** 2021-11-20T07:43:25.205 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:24 smithi112 conmon[42410]: debug 2021-11-20T07:43:24.893+0000 7fc83027d700 -1 osd.3 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:25.496 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:25 smithi112 podman[59410]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.3 2021-11-20T07:43:25.783 INFO:journalctl@ceph.osd.3.smithi112.stdout:Nov 20 07:43:25 smithi112 podman[59575]: 2021-11-20T07:43:26.458 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.3.service' 2021-11-20T07:43:26.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:26.498 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-11-20T07:43:26.499 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-11-20T07:43:26.499 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.4 2021-11-20T07:43:26.784 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:26 smithi112 systemd[1]: Stopping Ceph osd.4 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:26.784 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:26 smithi112 conmon[47260]: debug 2021-11-20T07:43:26.667+0000 7fee03975700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:26.785 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:26 smithi112 conmon[47260]: debug 2021-11-20T07:43:26.667+0000 7fee03975700 -1 osd.4 59 *** Got signal Terminated *** 2021-11-20T07:43:26.785 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:26 smithi112 conmon[47260]: debug 2021-11-20T07:43:26.667+0000 7fee03975700 -1 osd.4 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:27.209 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:26 smithi112 podman[59732]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.4 2021-11-20T07:43:27.534 INFO:journalctl@ceph.osd.4.smithi112.stdout:Nov 20 07:43:27 smithi112 podman[59896]: 2021-11-20T07:43:28.120 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.4.service' 2021-11-20T07:43:28.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:28.158 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-11-20T07:43:28.159 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-11-20T07:43:28.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.5 2021-11-20T07:43:28.435 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 systemd[1]: Stopping Ceph osd.5 for 16910ea2-49d4-11ec-8c2c-001a4aab830c... 2021-11-20T07:43:28.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 conmon[52158]: debug 2021-11-20T07:43:28.313+0000 7f367403b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T07:43:28.436 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 conmon[52158]: debug 2021-11-20T07:43:28.313+0000 7f367403b700 -1 osd.5 59 *** Got signal Terminated *** 2021-11-20T07:43:28.437 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 conmon[52158]: debug 2021-11-20T07:43:28.313+0000 7f367403b700 -1 osd.5 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T07:43:28.783 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 podman[60053]: ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c-osd.5 2021-11-20T07:43:29.283 INFO:journalctl@ceph.osd.5.smithi112.stdout:Nov 20 07:43:28 smithi112 podman[60217]: 2021-11-20T07:43:29.782 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-16910ea2-49d4-11ec-8c2c-001a4aab830c@osd.5.service' 2021-11-20T07:43:29.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T07:43:29.823 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-11-20T07:43:29.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T07:43:34.349 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T07:43:39.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:43:39.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T07:43:39.850 INFO:tasks.cephadm:Archiving crash dumps... 2021-11-20T07:43:39.854 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/ubuntu@smithi071.front.sepia.ceph.com/crash 2021-11-20T07:43:39.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T07:43:39.911 INFO:teuthology.orchestra.run.smithi071.stderr:tar: /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T07:43:39.912 INFO:teuthology.orchestra.run.smithi071.stderr:tar: Error is not recoverable: exiting now 2021-11-20T07:43:39.916 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/ubuntu@smithi112.front.sepia.ceph.com/crash 2021-11-20T07:43:39.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T07:43:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T07:43:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2021-11-20T07:43:39.967 INFO:tasks.cephadm:Checking cluster log for badness... 2021-11-20T07:43:39.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2021-11-20T07:43:39.999 INFO:teuthology.orchestra.run.smithi071.stderr:grep: /var/log/ceph/16910ea2-49d4-11ec-8c2c-001a4aab830c/ceph.log: No such file or directory 2021-11-20T07:43:40.001 INFO:tasks.cephadm:Compressing logs... 2021-11-20T07:43:40.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T07:43:40.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T07:43:40.069 INFO:teuthology.orchestra.run.smithi071.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T07:43:40.071 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T07:43:41.185 INFO:tasks.cephadm:Archiving logs... 2021-11-20T07:43:41.186 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/ubuntu@smithi071.front.sepia.ceph.com/log 2021-11-20T07:43:41.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T07:43:41.507 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/ubuntu@smithi112.front.sepia.ceph.com/log 2021-11-20T07:43:41.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T07:43:41.688 INFO:tasks.cephadm:Removing cluster... 2021-11-20T07:43:41.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --force 2021-11-20T07:43:42.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 16910ea2-49d4-11ec-8c2c-001a4aab830c --force 2021-11-20T07:43:42.706 INFO:tasks.cephadm:Removing cephadm ... 2021-11-20T07:43:42.707 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T07:43:42.725 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T07:43:42.742 INFO:tasks.cephadm:Teardown complete 2021-11-20T07:43:42.743 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-11-20T07:43:42.761 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-11-20T07:43:42.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T07:43:42.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T07:43:42.859 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-11-20T07:43:42.872 INFO:teuthology.task.clock:Checking final clock skew... 2021-11-20T07:43:42.872 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T07:43:42.876 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T07:43:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2021-11-20T07:43:42.891 INFO:teuthology.orchestra.run.smithi071.stdout:210 Number of sources = 4 2021-11-20T07:43:42.891 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T07:43:42.891 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2021-11-20T07:43:42.892 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 39 -4887us[-4887us] +/- 117ms 2021-11-20T07:43:42.892 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 2 6 377 40 +431us[ +423us] +/- 56ms 2021-11-20T07:43:42.892 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +658us[ +658us] +/- 63ms 2021-11-20T07:43:42.893 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 38 +1741us[+1741us] +/- 76ms 2021-11-20T07:43:42.893 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2021-11-20T07:43:42.895 INFO:teuthology.orchestra.run.smithi112.stdout:210 Number of sources = 4 2021-11-20T07:43:42.895 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T07:43:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2021-11-20T07:43:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 40 -4934us[-4902us] +/- 117ms 2021-11-20T07:43:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 2 6 377 39 +394us[ +426us] +/- 56ms 2021-11-20T07:43:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +621us[ +621us] +/- 63ms 2021-11-20T07:43:42.897 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 39 +1701us[+1701us] +/- 76ms 2021-11-20T07:43:42.897 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-11-20T07:43:42.911 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-11-20T07:43:42.912 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-11-20T07:43:42.936 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T07:43:43.005 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T07:43:43.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T07:43:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:43:43.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T07:43:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:43:43.120 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-11-20T07:43:43.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-11-20T07:43:43.169 INFO:teuthology.task.internal:Duration was 1207.076017 seconds 2021-11-20T07:43:43.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-11-20T07:43:43.198 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-11-20T07:43:43.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T07:43:43.202 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T07:43:43.243 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T07:43:43.245 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T07:43:43.544 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-11-20T07:43:43.545 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2021-11-20T07:43:43.545 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T07:43:43.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2021-11-20T07:43:43.576 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T07:43:43.597 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-11-20T07:43:43.598 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T07:43:43.619 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T07:43:43.688 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2021-11-20T07:43:43.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T07:43:43.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T07:43:43.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-11-20T07:43:43.859 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-11-20T07:43:43.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T07:43:43.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T07:43:43.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-11-20T07:43:43.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T07:43:43.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T07:43:43.956 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2021-11-20T07:43:43.978 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2021-11-20T07:43:43.991 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T07:43:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:43:44.026 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T07:43:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T07:43:44.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-11-20T07:43:44.063 INFO:teuthology.task.internal:Transferring archived files... 2021-11-20T07:43:44.068 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/smithi071 2021-11-20T07:43:44.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T07:43:44.128 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515123/remote/smithi112 2021-11-20T07:43:44.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T07:43:44.178 INFO:teuthology.task.internal:Removing archive directory... 2021-11-20T07:43:44.178 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T07:43:44.181 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T07:43:44.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-11-20T07:43:44.243 INFO:teuthology.task.internal:Not uploading archives. 2021-11-20T07:43:44.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-11-20T07:43:44.286 INFO:teuthology.task.internal:Tidying up after the test... 2021-11-20T07:43:44.287 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T07:43:44.290 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T07:43:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: 262155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 20 07:43 /home/ubuntu/cephtest 2021-11-20T15:34:08.207 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-11-20T15:34:08.244 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2021-11-20T15:34:08.246 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2021-11-20T15:34:08.246 DEBUG:teuthology.exit:Finished running handlers